Debounce API Requests in a JS Class

I have a react application that needs to access an API endpoint on the client side - which is hit when a user clicks on a button, as well as a few other places in the UI. The trouble is, the API endpoint is a cloud function, which is charged on a per-execution basis.

I don't want users to be able to spam-click this button and have to pay for it, but I also don't want to disable the button during the request - which would encourage the user to view the source and spam my API themselves. I want something to work invisibly, so they think that everything is working as they expect - a new request every time they click.

Because of how I crafted the API response, only the last request really matters within a specified amount of time, which means a debounced API request function would be perfect.

Here's an overview of how I implemented it, based off of the lodash debounce method:

Mass Effect Checklist

Check it out →
GitHub repo →

I wanted to build something for my portfolio, so I decided to over-engineer something basic with a sketchy idea: the Mass Effect Checklist! 😃


Mass Effect is by far my favourite game series, I've played through the game so many times that I could just-about recite everything by heart.

To aid in my unhealthy addiction, I created a list of things that I need to do - allowing me to get to the ending that I like the most. Sometimes I forget what I'm supposed to do or how to do something, and end up having to go over to the Wiki. The problem is, the Wiki is hard to follow. I felt like this checklist format is easier and more logical.


It's currently at something of an MVP stage, and at the time of writing, I haven't even finished typing-up all the missions. At the moment it's just a proof-of-concept.

I've recently forced my friend to finally experience the game, so he's my test guinea pig. However, I would love it if more people gave it a go, so send my some feedback by contacting me here after using it for a bit.

Future changes will mostly be on UX, interactivity (such as a success calculator for certain missions), a timeline to show an overview of the list, syncing the checklist to the cloud, and maybe a saved-game upload functionality - to check the list based on your progression, check success requirements for missions before you even get to them, and show some stats that are normally hidden in the game.

Technologies used

  • React
  • Netlify
  • Foundation
  • Azure Functions
  • Application Insights

Check it out →
GitHub repo →

Notes on Cloud Providers Offerings for Serverless

I just had some notes I wanted to share regarding using serverless functions on a couple of different cloud providers.

Microsoft Azure

  • My go-to provider for functions
  • Supports secrets for things like connection strings
  • Can be configured to use inputs and outputs from Azure services, already authenticated
  • Works best with functions written in C# (non-crx)
  • C# script (crx) is not really supported in IDE's, mostly to do with importing libraries
  • Node.js and Python are still not quite mature yet
  • C# projects have the fastest cold-startup, while Node.js is the slowest
  • Python is always slow to run regardless, because it's run through some type of virtualised environment that you have to set up yourself
  • Good luck using Python wheels or pip
  • Function runtime is based on input and output - if for example you send an output, the function will be terminated soon-after, rather than letting the function terminate in code. This makes eventual data design impossible.

Google Cloud

  • Node.js functions only
  • Slightly slower to run overall, but doesn't appear to have a cold-startup that I could see
  • Function runtime is NOT based on input and output, functions can terminate programmatically
  • Secrets storage, like environment variables, are not supported
  • Git integration was impossible for my workflow for the above reason, had to paste into the web editor with my secrets in plaintext.
  • Google Cloud libraries are automatically imported, and already authenticated based on the account running the function

Steganographic Message Extraction from Image

Steganography is the practice of hiding data in plain sight, such as within the pixel data in an image, video, or audio file. There are infinitely many ways of doing this, by changing the order or distance that the message bits are stored from each other - but it is most important that it remains constant, and that the person you're sending the message to knows how to decode it!

I have a snippet for you to do the decoding. This snippet assumes you have an image where a pascal-type string message has been stored in row-major, least-significant-bit fashion.

This snippet requires PIL, or its modern fork - Pillow (pip install Pillow)

HTTP Authentication Dictionary Attack

Here's a little snippet of Python code to crack HTTP digest authentication, getting a user's password given information from a captured packet and a wordlist.

You can find some good wordlists right here.

Sharing WiFi Connection over Ethernet Ubuntu 18.04

I found my Raspberry Pi in a drawer, and decided to get back into developing small IoT applications for it. Most of what I wanted to do required access to the internet, and my USB WiFi adapter is super trash.

The good news is that Pi's have an RJ45/Ethernet port, the (sorta) bad news is that my house is WiFi only.

To address this, I went wanted to share my Ubuntu 18 desktop's WiFi connection over Ethernet - however, some of the software you need is no longer readily available, as it is superseded by the new settings app in Ubuntu 18.

What you're after is available through the terminal, with:

nm-connection-editor

When it opens, select the wired connection item, clicking the edit button (the cog). In that menu, switch to the IPv4 tab, and select the method: 'shared to other computers'.

Screenshot of the connection editor with settings opened.

After that, save everything and connect your cable if you haven't already, and DHCP should kick-in and set everything up for you!

You could even connect a switch or hub and share the connection further, if you wanted.

Note that if you need to get the IP address of the connection, you can use ifconfig. You'll only need this if DHCP doesn't automatically configure everything.

Feelin' Retro

And by retro, I mean something like circa 2012, because this is gonna be another meta post about my blog / blogs in general. It's like this blog is purely about my blog at this point 😆

I was looking at my home page and was thinking, "my, these gaps between things are much too large". Yes I am pretty much the only source of traffic 'round here, but that's not the point, the point is that I snapped and decided I wanted something simple. With navigable negative space!

Thankfully, there's a Blogger theme with exactly that name, and here we are. It, like everything about this CMS, is clearly ancient, but I'm totally feeling it at this point in time.

I could go on about the unpleasantness of modern web design to fill-out this post, but I'm too tired, so I'll spare you for now.


Enjoy your day 👍

Alternative Backup Locations

I have a few Ubuntu machines here-and-there, and one of my most recent additions is my MacBook Air. I installed Ubuntu 17.10.1 and so far it's okay.

One thing that was causing me some grief, however, was the Deja Dup update. This update to the default backup software interface left me without the option to backup to Google Cloud Platform's blob storage service.

While there was still the possibility that the underlying CLI software (Duplicity) that powers the front-end (Deja Dup) may still have this functionality available, I wasn't about to go down that route.

The solution for me was to provision a networked file drive through Microsoft Azure. This allowed me to mount a drive at login through fstab and set a scheduled backup like I would do normally.

Since most of my files and media exist in the cloud anyway - by way of Google Drive, GitHub, Netflix, and Apple Music - I only needed a small 250GB file drive. This drive can also be shared with other devices, if you like to live on the edge of increased risk of data loss / don't want to set up another drive, like myself.

Screenshot of my network drive backup location






The pricing isn't too bad, especially considering that you only pay for what you use. If you find your bill is getting too high, set the software to delete old backups (avoid deleting files yourself, you might delete something important!).

Of course, you don't have to use Azure. You could use DigitalOcean, for example, and set up a droplet to host some block storage over a samba share.

 

Pro tip

You'd do yourself good by encrypting your backups with a unique password, since this is a network drive, so you should always assume that a hacker could get access. The encryption feature is available during the first backup.

Also note that an encryption password won't stop some asshole from deleting all of your backups, leaving you in a precarious position. Make sure you have a periodic offline backup of all your important files, even if you have to do it manually.

Eventual Data Design

I do a lot of work with web API's, particularly as React web-app back-ends, and an issue I constantly run-into is the speed at which these requests return. These long-running web requests makes pages slow to render, with content having to be hidden behind a loader or other skeleton animation while the user waits.

These React projects, however, are low-traffic and low-value. I don't want to have to pay for servers to continuously pre-cache data that's nice for a user to have and access quickly, but isn't all that critical or value-adding to the system overall.

While thinking about this, I was wondering why I couldn't just do the long-running processing for the data after the request had terminated? I was working in C#, which is well known to separate the actual application from the web server - meaning that you can't control the web server from within your app in a very meaningful way. To me this meant a change in technologies, to something where the web server was actually part of the application - this was a job for Node.js.

In Node.js, you don't have to return a script to terminate the web request. This is perfect for what I had thought about.

 

The design outline

  1. The user hits the JSON API, and they are either returned initial data displaying default values, or data from the cache.
  2. In the JSON object, a field describes the state of the cache - if the data will be renewed after this request, when the data was last cached, etc., so that the client knows how old the data is and can make decisions based on that.
  3. After the request is terminated, the timestamp on the cache is evaluated to determine if the data needs to be refreshed (or is ignored if it always needs to be as up-to-date as possible), and then the script continues to do the typical things that you would do to process or otherwise add value to the API data.
  4. Transformed data should be added to a fast database, such as MongoDB, Redis, or DocumentDB, so it can be looked up quickly in the first step.
  5. Script terminates.

 

Pros

  • Data is consistently quickly delivered to the user
    A simple key/value query to a database is much faster than selecting information from a range of data stores, and then processing it. The transformations applied to different sets of data may vary in the amount of time it takes to complete the job, meaning if this job occurs while the client is waiting, they may cancel the request before it is complete.
  • Longer-running data transformations can be employed
    Data is only ever processed when the client isn't waiting for a response, meaning a longer-running query or data transformation process can be applied without slowing down requests. This results in more valuable data being delivered more consistently.
  • Cost of data processing is dramatically reduced
    The data is only processed when the client requests it, rather than having to run a timed or constant job to keep data up-to-date. It is impossible to predict exactly when a client is going to require this data, so it would be wasteful and expensive to continuously update these sets of data when it's only going to be requested a couple of times per day by each client - perhaps not at all. Machine learning could be applied to discover when a client is most likely to make a request, but this is adding more cost in an attempt to lower costs.

 

Cons

  • Data is always outdated
    This design shouldn't be applied to critical data sources, as the information being displayed will always be tied to when the job was last performed for this data set. The issues is that it could be at best a few seconds old, or at worst months or even years, depending on the client's activity.
  • Repeat requests defeat the purpose
    Depending on your design, you may run-into issues where the client hits the API multiple times, before the last requests have completed their work. This means that the potentially expensive data transformation may occur multiple times, with only one transformation being value added - a sort of race-condition. This can of course be fixed with ACID transactions, but it's always going to be a possibility while the work dispatcher is controlled by the client (as they initiate API requests), and not an orderly scheduling system.

 

Example

If you want to see an example of how I implemented this with Google Cloud Platform's Functions, I have a now-defunct set of scripts located on GitHub: interflare/archived-functions.

There are of course many improvements than can be made to prevent repetitive requests, such as a system that marks the job as in-progress in a database, but with the project I was working on, it didn't really matter.

 

Conclusion

This design probably isn't new, but I believe it's a great way to think about accessing and processing data that means little to clients and your application, but is nice to have nonetheless. Rather than stripping out everything that isn't going to make you money, this might be a dirt-cheap way of distributing supplementary data that could mean a customer chooses you over every other service.

Kali Linux on DigitalOcean

Kali Linux is a distribution of Linux that is used for penetration testing. While normally distributed in a typical ISO file for install on hardware, it has also been wrapped into a Docker image to run it on basically any OS with Docker installed.

On DigitalOcean, you can't install an ISO file. This is where the Docker image comes into play. On DigitalOcean you can set up a container distribution to use Kali. I like to use CoreOS.


Once you've set up your container distribution, go ahead and login with your SSH key, using the 'core' user. That was one thing that was not immediately obvious, because I would typically SSH into virtual machines with 'root'.

As the Kali Docker documentation explains, you can pull the OS like so:

And that's it! You can install and update the tools you'll need as you would normally.


It's important to note, you can't install the GUI version of Kali Linux like this. If you wish to do that, you will need to set it up as a guest in something like OpenVZ or Xen, and then connect to it through VNC or RDP.

One-Page App Routing on Netlify

While playing around with react router on a react app I'm building and deploying to Netlify, I noticed that I would get a 404 page not found error if I fully-refreshed the page, or entered from a location other than the index.

Looking deep into the Netlify docs I found out that you have to add a little file to your /public directory named '_redirects' with the following content:

All it does is rewrite all requests to any file that doesn't already exist to the index page, where react router can handle it.

Note that navigating to a file that exists from the app will not work without a forced refresh. For example if you were at '/', and you typed '/file.png', you would be passing '/file.png' to the router, of which a route (probably) won't exist.

Pretty simple!

Here we go again with Blogger 🙄

I decided I wanted to switch to a static blog from Blogger a few months ago, mostly because of the performance benefits of plain-old HTML. I don't know what's happened to me or Blogger since then, but this platform has somehow managed to pull itself out of the gutter and into my life as my blogging home once more.

Blogger obviously has more pro's than the likes of Jekyll or Hugo or *insert static generator here*, an actual editor and dynamic fixtures for starters, but the reason I originally ditched it was because of its difficulty regarding customisation and, most importantly for me, page load times.

First of all, customisation on Blogger is hell. The only thing you get to edit is this massive, confusing, super-dense, and non-standard XML file. Yes, you get that theme customiser where you can dump some CSS and turn a few knobs. Fine. I wanted to set-up things like Twitter cards per page/post in the meta of the HTML, but it turns out that it's actually impossible with the way that Blogger was designed. This is still the case today, and it's still a push factor.

The second push factor, as I said, was the page speed. My original Blogger home page used to take about four seconds to load, according to previous speed tests. I hated this, not that it really matters because this blog is obscenely obscure, but it was super embarrassing to me. This is what tipped the scales for me, and I ended-up moving my posts over to GitHub pages. I didn't really have much I wanted to move, so there was definitely a tonne of archive 'shrinkage'. Actually, now that I think of it - I didn't move any posts. I reckon I was glad to have the excuse to erase my dumb stuff.

But yeah, I'm back on Blogger because I felt like the struggle of: creating a new markdown document, typing-up all the front matter, and then pushing to git, was just such a drag. Once I got back up and running, I actually noticed that things were running much faster than I remembered. I don't know if it's because I'm publishing this using a G Suite account this time around - maybe there's a different set of servers or framework for paying customers.

Regardless, I actually think I missed this ugly WYSIWYG editor. At least I don't have to type-in the damn date and time now.

SSL on Blogger with Custom Domain

[Update 2018-06-27]: Wow okay, Google really dusted-off the Blogger codebase and decided to implement SSL on all custom domains now. They must have seen my post and decided to implement it ;) that's my #IMPACT.

---

Once again, I changed my mind with which service I want to use for my blog. This time around, I decided to go back to Blogger - a dynamic blogging service, as opposed to my previous static blog. However because Blogger is getting a little bit old now, it's a bit of a pain in the ass to implement the expectations of a modern website - particularly SSL.

SSL is managed and enabled by Google by default, if you use their subdomain. If you want your own custom domain, they expect you to go without SSL. However, this can be fixed with our good 'ol pal CloudFlare.


Go ahead and add your Blogger domain to CloudFlare like you would normally, and do the same on Blogger.


Note that you might have to 'grey-cloud' your domains on CloudFlare while you verify them on Blogger, but make sure you switch them back to orange when it's done, or the reverse proxy won't be enabled.

Then head over to the 'SSL' section of CloudFlare, and enable the following:
  • Always use HTTPS
  • Require modern TLS
  • Automatic HTTPS rewrites
It can take up to 24 hours for your CloudFlare certificate to be issued, so don't expect it to work right away. However after that period of time, you should have a tidy green padlock for your new blog!

Let me know in the comments if you're having trouble, and I'll help you out.

Why I dumped Linux for Windows

For most developer types like myself, it’s usually the other way around - ditching Windows for Linux. However, I find myself in an interesting position.


My original reason for switching over to Linux a year or two ago, was because I was fed-up with Windows 10’s sluggish performance on my otherwise decently powerful desktop. I dual-booted Ubuntu 17.10, and it worked pretty well - switching back to Windows occasionally when I wanted to play video games or use Visual Studio. Ubuntu was responsive as my daily driver, it was easy to develop on, and most importantly it was free.

Over time I noticed things start to slow down. The performance issues were becoming more and more obvious, but they were not the same as my issues with Windows, where it was slow from day one. I was thinking that I was bloating Ubuntu with all the nonsense I was putting on it - all sorts of virtualisation programs and heavy disk encryption that probably wasn’t necessary.

This was when I switched to Linux Mint, just at the start of this year - literally twenty days ago. I installed it in the same way as I did when I originally switched to Ubuntu - just threw in another SSD and dual-booted with my current system, so I could bring all my old documents a little easier.

I was surprised that a new install of Linux Mint was as slow as Windows, but of course it wasn’t actually the OS’s fault. The SSD I used was the same one that I had previously installed Windows. So it was this ageing SSD’s fault all along!

So I gave Windows another shot on a newer Intel SSD, and I was absolutely blown away with the speed. It was faster than Ubuntu on an identical Intel SSD! Login time was near-instant, and programs were loading faster. Of course this is all based on my perception, I didn’t actually time or test anything. I’m probably being tricked by partial loading or that skeleton loading cleverness.

Nevertheless, the responsiveness of the recent Windows makes me think that the reasons for anyone actually making the switch to Linux are quickly diminishing.

Some bullet points

  • Speed
I’ll give Linux the advantage of a lower overhead, but otherwise Windows appears to beat-out Debian Linux in performance, in my experience. Perhaps you’ll see greater reason for Linux on less powerful or old hardware.
  • Compatibility
I’ve never had to worry about any drivers other than for my old printer on Windows. They have everything ready for me during installation, whereas on Linux I’ve always had to install my own drivers or connect to ethernet to get WiFi up and running. There must a trade-off with software/driver bloat or licensing that I’m missing, but I clearly prefer the direction Windows went.
  • Software
Windows has almost everything. I’ve been known to play a few video games in my spare time, and the only Linux-native games I can play are indie 2D-sidescroller games. And then of course there’s the Adobe suite, iTunes, and Visual Studio - all of which are non-existent on Linux. And if ONE PERSON tries to recommend ‘TuxPaint’ or ‘TuxTunes’ or any of those other ugly, poorly designed alternatives, I will cry. It’s 2018, we polish our software visuals now - even if it’s FOSS.
  • Security
I was debating this argument internally, but I think it works here. I know Windows has a bad rep for getting infected with all sorts of viruses, but at least it has proper defences available. The defence strategy for Linux is obscurity, which is not a defence strategy at all. And perhaps people confuse correlation for causation - the weakest link in computer security is always people, and people who use Linux are likely to be computer savvy.

Your parents, grandparents, and children are likely to be using Windows because of its prevalence, and these are the people the download and run these random programs because and advertisement told them to. I can bet you that the same thing will happen on Linux, if it was the most popular OS. The difference is that Windows has specialised software like Malwarebytes and Avast to remove and prevent infection, where on Linux you would have to restore to a segregated backup, or just re-install the OS. Maybe you would remove it manually, if you knew what you were doing. // end security rant.
  • Price (??)
This one was interesting for me as well. Microsoft is taking something of a gentle approach with payment for using their software these days. You can download an ISO off their website for free, and use it for quite a while without registering it. I’m currently running the OS like this, and all it’s restricting me from doing is customising my desktop and task bar. It also has a little watermark in the corner. I can continue to use it like this until uni starts again and I can get my free student license, or when my friend stops using the home license I purchased a while ago - whichever happens first.

Yes, I know that Linux is always free, but I would argue that on this front that Linux and Windows are on a largely even playing field, given the features and support of Windows.



Hopefully that didn’t cause too much cognitive dissonance, but that’s how I’m feeling about the current state of operating systems. What a geeky thing to think about!

Distribution of Circular Queued Work in Node.js

I was building an application that worked with Twitter for a friend, but it unfortunately got killed by Twitter for violating their terms of service - oops. However, that application, while it was running, needed to be able to work with a lot of inputs at critical times. There was no way that a single operator on a circular queue, slowly working its way through linearly, was going to cut it - more and more items would expire before it can get back to the beginning.

Instead, what I decided was best was to simply add all of these work items into a database, and execute some interesting queries on child processes - ‘workers’.

If you don’t want an explanation, I put together something of an example on GitHub, right here.

 

The workers

I didn’t bother with anything too fancy - even though I definitely did over-engineer this example. It’s just a forked script with IPC for communication, sending each other information via objects.

 

index.js

The index.js file is our root script - the parent. It creates worker processes and adds it to an array, that allows us to interact with it from the parent.

Node.js sets up IPC (inter-process communication) automatically, and we send the worker its ID. I chose to set an ID myself in the parent, but you could just as easily use the workers process id, as it would be better scoped to the server.

I also set up an ‘awaiting payload’ system, so that we could setup health checks and dynamically drop and spawn new processes. I haven’t actually developed this yet, but maybe later.

The setInterval at the bottom of this script it to keep the script running forever - otherwise it would terminate as soon as it reached the bottom of the script, and kill the forked processes before they can even do anything. If you were using this script in a framework such as express or sails, this is unnecessary as they keep the script running anyway.

Fun fact! - A javascript interval is never scheduled when it is set to infinity, even though the script expects it to do so, meaning the script never terminates.

 

worker.js

Pretty basic! All it does at this point is receive some info from the parent, and schedule a work loop to run every second.

This requires the worker loop to get the data itself, rather than relying on the parent to send it data to work on.

The thinking behind this type of design is that the worker can get the data to work on as it is required. There’s never a moment when it has nothing to work on, or where it has too much to work on.

This design, however, requires some slightly fancy queries to select data from a database (mysql in this case) in a way that prevents race conditions and over-working on the same work item.

 

The SQL queries

All of the following queries would be placed into the work_loop function in worker.js, using your favourite MySQL connector or equivalent. You can see this in action in the GitHub example I’ve set-up.

In any other project, you would probably begin by SELECT-ing work items that haven’t been worked on, then UPDATE-ing to claim a few items as being worked on by a particular worker process. This won’t work, they will no doubt select the same rows. You might also think to use a transaction, but this probably won’t cut it either.

What we need is something atomic - completing in a single query. The simplest way to do this is to flip our queries around. UPDATE first, SELECT second. How trivial!

With the first ? param being the ID of the worker.

As you can see, this selects the work item that is the oldest, and either unclaimed, or where the claim is old enough that it was likely to have stalled or failed.

You can then easily select the work based on the worker’s ID, perform the work, and then unclaim the work! Easy as!

Linux Mint Development Environment Setup Quirks

As part of my ‘new year, new me’ campaign for 2018, I decided to switch from Ubuntu 16.04 to Linux Mint 18 - can you guess why?

 
I don’t have that much to set up most of the time, save for some development tools such as VS Code and GitKraken. I otherwise use the default applications for everything.

After admiring the Cinnamon desktop environment for a little while, I went about installing my development tools. I came across a few issues, mostly to do with the different distros shipping with different software.

Here are some of these problems, with solutions.

Encrypted 7zip files

Since I was moving operating systems, I took some backups of the various folders and other things I wanted to take with me.

I had to put them somewhere other than the drives I was about to wipe, and they were too large for any thumb drive I had lying around, and it would take far too long to upload to Drive. My only option was to put it on a public samba share in my home network, obviously having to encrypt them.

When I got my new install of Linux Mint connected to the network share, I found I couldn't decrypt the 7zip files with the default archive manager.

Problem

The issue was that I could not open the files, because the p7zip package does not ship with Linux Mint, whereas it must obviously ship with Ubuntu.

Solution

Simply add the package with apt:

sudo apt install p7zip-full

And then try again with the archive manager, should be all good.

GitKraken not starting

I got into GitKraken because they let you use the pro version for free if you're a student, as part of the GitHub education pack. It's pretty great, and very good looking. If it weren't my daughter I would probably be dating it.

Anyway, it installed on Mint just fine with a deb package, but it failed to open.

Problem

It appears that it requires libcurl3 to be installed, and - like last time - doesn’t seem to ship with Linux Mint.

Solution

Once again, super easy. Just hit it up with the terminal:

sudo apt install libcurl3

And you should see that ugly squid the next time you try to launch it.

Featured

Debounce API Requests in a JS Class

I have a react application that needs to access an API endpoint on the client side - which is hit when a user clicks on a button, as well as...

Popular