Creating with Data

Why Some Code Examples Are No Longer Working

Rory Gianni — Mon, 04 Aug 2025 10:08:25 GMT

Some of the interactive code examples and demos in older posts on Creating With Data were originally hosted on Glitch, a platform for building and sharing live code projects.

Recently, Glitch announced that it is shutting down its free hosting services. As a result, many of the embedded demos and linked projects that relied on Glitch may no longer load or function correctly.

I’m currently working on migrating this blog and all the projects I had on Glitch, including my personal website.

I'm going to consolidate many of my personal sites and projects, but it'll probably take me a while. It's quite an undertaking as I've been using Glitch for about 8-9 years now!

Saving D3.js animations as video files with WebAssembly

Rory Gianni — Tue, 29 Oct 2024 17:28:57 GMT

Browsers are more powerful than ever! Non-browser software can be compiled to WebAssembly and used client-side. WebAssembly or 'WASM' acts as a bridge, allowing code written in languages like Rust, C, and C++ to execute smoothly in the browser.

In my SuperMap.world side-project I'm using D3 to generatively create lots of map designs in PNG and SVG formats. To up the challenge, I've been considering adding map animations using d3-transition.

The only problem is compiling video can be a pretty heavy operation (certainly for servers on a side-project budget). Thanks to WebAssembly, we can now get those lazy freeloading clients to do some of the work.

To tip my toe in the water, I've been experimenting with Resvg and FFmpeg (compiled to WASM) to see if I can capture D3.js transitions and render them out to Gif, MP4, or whatever format – no servers required!

Before you ask – yes, I know that rendering to Webcanvas in a raster format then capturing with FFmpeg makes more sense but I have my reasons which I cannot be bothered to explain right now.

Capturing PNG with Resvg

Resvg is a powerful tool for rendering and working with SVGs. I've been using Resvg to convert SVG to PNG from the command line and it does a great job. The plan here is to use it to capture frames that will eventually be compiled into a video.

If you want to try it out on its own all you need to do is import and write a capture function e.g.

async function captureSvg() {
    // Initialize Resvg WASM
    await resvg.initWasm(
        fetch('https://unpkg.com/@resvg/resvg-wasm/index_bg.wasm')
    );

    // Get the SVG element and convert to string
    const svgElement = document.getElementById('mySvg');
    const svgString = new XMLSerializer().serializeToString(svgElement);

    // Create Resvg instance
    const resvgJS = new resvg.Resvg(svgString);

    // Render to PNG
    const pngData = resvgJS.render();
    const pngBuffer = pngData.asPng();

    // Create blob and download link
    const blob = new Blob([pngBuffer], { type: 'image/png' });
    const url = URL.createObjectURL(blob);
    
    // Create download link
    const a = document.createElement('a');
    a.href = url;
    a.download = 'capture.png';
    a.textContent = 'Download PNG';
    document.body.appendChild(a);
}

As you'll see above it goes and fetches a .wasm file. Sometimes these files can be quite large – FFmpeg is about 30MB!

In the above example we get a PNG version of the SVG and a download link. We don't want to download a single PNG, but the above is a concise SVG to PNG convert function (NB if you want text rendered correctly you'll need to hand that to Resvg).

Combining D3 + Resvg + FFmpeg

What we want to do is a little different: capture a PNG every time something changes, (e.g. inside a D3 tween function), hand that PNG over to FFmpeg to use as a frame in the video. Once we're done animating we compile and offer the MP4 up for download.

This example animates a simple SVG circle and captures each frame. Here's the code for this example (available on Glitch):

Result

0:00

The capture queue

The real magic here happens in the captureFrame function. We want to write multiple frames to FFmpeg's internal file handler, as a sequentially numbered file (e.g., frame_00000.png, frame_00001.png).

We use captureQueue to manage asynchronous frame capturing. Chaining frames in a promise queue ensures each frame finishes processing before moving to the next.

This approach resolved an issue where D3’s tween function wasn’t running synchronously, leading to duplicated or misordered frames.

// Frame capture function
let frameCount = 0;
let captureQueue = Promise.resolve();
async function captureFrame(svgString) {
  captureQueue = captureQueue.then(async () => {
    const resvgJS = new resvg.Resvg(svgString);
    const pngData = resvgJS.render();
    const pngBuffer = pngData.asPng();

    const filename = `frame_${frameCount.toString().padStart(5, '0')}.png`;
    console.log(`Capturing ${filename}`)
    await ffmpeg.writeFile(filename, pngBuffer);
    frameCount++;
  });
  return captureQueue;
}

Using captureFrame is simple. We just call it whenever we want to add a frame, such as in a d3 tween function.

Frame rates

When it comes to D3, it's not possible to dictate frame rate. From what I understand, D3 will aim to go as fast as supported by your browser. That's not always possible if you have complex transitions going on. As a result, the frame rate you tell FFmpeg to use may not match what's happening in the browser. I imagine you could track your animation's FPS yourself and pass that to over to FFmpeg at compile time.

The GIF I rendered of the transatlantic cable visualisation has a lot more going on than the circle animation. It came out at about 6 or 7 FPS. That's fine for a GIF but not what you'd want from a "real" video.

A Gif version of the telegraph cable visualisation

Since these transitions use an interpolator I was able to render higher FPS videos by dropping D3 tweens in favour of just capturing renders inside a for loop –

// Use this  ...
const framesPerMillisecond = FRAME_RATE / 1000;
const frames = positionDuration * framesPerMillisecond;
for (let i = 0; i < frames; i++) {
  positionTween(i / frames, ip, sphere, land);
}

//Instead of ...
d3.transition().duration(positionDuration).tween("render", () => {
  return t => {
    positionTween(t, ip, sphere, land);
  }
}).end();

While my browser wasn't able to keep up (i.e. we can't watch the nice animation) it does succesfully produce a smoother video!

0:00

Choropleth mapping house sale prices - Part 2. Visualisation

Rory Gianni — Fri, 09 Aug 2024 14:04:20 GMT

In Part 1, I focused on readying the data for the visualisation. As far as the actual visualisation starter was concerned we got to the point of setting up the SVG with the boundaries projected. Now, we need to pick up our crayons and do some colouring in!

Before proceeding, take a quick look at the code for the full visualisation. You can view/remix the final code over on glitch. Also, if you haven't seen it already, go checkout the visualisation here.

In this post I won't go through each line but rather refer to the key steps that bring the visualisation together.

1. Import and prep the price data

The starter didn't use or refer to any of the pricing data we prepared, so we need to import it and ready it for binding to the geographic boundaries.

const prices = await d3.csv("https://creating-with-data.glitch.me/scothousing/intermediate-zones.csv");
const pricesMap = {};
const combinedPrices = prices.map((d) => { 
    d["combinedMean"] = combinedWeighted(+d["2021: Mean"], +d["2022: Mean"], 1, 2) * 1.033;
    d["combinedMedian"] = combinedWeighted(+d["2021: Median"], +d["2022: Median"], 1, 2) * 1.033;
    d["combinedLower"] = combinedWeighted(+d["2021: Lower Quartile"], +d["2022: Lower Quartile"], 1, 2) * 1.033;
    d["combinedSales"]  = (+d["2021: Number of sales"]) + (+d["2022: Number of sales"]);
    pricesMap[d["Feature Identifier"]] = d;        
    return d;
});

This code loads the sale data from a CSV file and calculates a weighted average for house prices across 2021 and 2022. The combinedWeighted applies a slight adjustment (* 1.033) to represent inflation to date (based on inflation stats). This data is then stored in a pricesMap object for quick lookup by geographic area.

1.1 Adding controls

We have a few different columns of data for the user to choose from, as well as setting their 'amount available'. The following controls in the markup are referred to by d3 later on to help choose values from the pricesMap object.


    
        Lower quartile             
        Median 
        Mean         
    

    Amount available

There's not a lot to this, just plain HTML radio inputs and a slider. We'll refer to this selection later in the render function.

2. Colouring the boundaries

The main magic happens in the render function. It's a long one, so let me break it down.

2.0 Reading user selection

The function starts by checking which radio button the user has selected, and keeping a track of that in selectedColumn.

const selectedColumn = d3.select("input[name='column']:checked").node().value || "combinedLower";

2.1 Calculating the colour scale

Once the relevant data column is identified, the function extracts the house price values and calculates min and max values. These define the range for the colour scale using D3's scaleSequential and interpolateCool functions.

const values = combinedPrices.map(d => d[selectedColumn]);
const minMax = d3.extent(values);
const color = d3.scaleSequential(minMax, d3.interpolateCool);

The resulting 'color' here, is a function that when passed a number, will turn that number into a colour value.

2.2 Rendering the legend and boundaries

We first use the color function when generating the legend. This appears as a little gradient at the bottom right of the visualisation. The number of steps are controller by the numIntervals loop:

const intervalValues = [];      
for (var i = 0; i < numIntervals; i++) {
    intervalValues.push(i * (minMax[1] / (numIntervals - 1)));
}

legendColours.selectAll("div")
    .data(intervalValues)
    .enter().append("div")
    .attr("style", function(d, i) { 
    return "background: "+ color(d) +"; width: " + lWidth + "px"; 
})

legendNumbers
    .selectAll("div")
    .data(intervalValues)
    .enter().append("span")
    .text(function(d) { return "£" + Math.round(d); });

const areaPaths = areasGroup.selectAll("path")
    .data(geoFeatures.features)
    .enter()
    .append("path")
    .attr("data-centroid", (d)=>{ return JSON.stringify(pathDefiner.centroid(d)) })
    .attr("d", pathDefiner)
    .attr("title", (d) => { return d.properties.Name })
    .attr("id", (d) => { return "g-" + d.properties.InterZone; })

After rendering the legend, we render each region as a path element. Attaching some data, title, and id attributes acts as a handy reference for further interaction and debugging.

2.3 Applying colours to the boundaries

We know what column the user wants to visualise, and we know how to turn the price values into a colour. The updateFill is responsible for using the color function and applying it to each geographic area (path element) on the map.

Our visualisation involves more than just visualising the chosen price value for each region, there's also the "amount to spend" in the mix too.

const updateFill = () => {
  const spend = (amountInput.node().value / 100) * maxAmount;
  amountDisplay.text("£" + Math.round(spend));

  areasGroup.selectAll("path")
    .attr("data-value", (d) => { return pricesMap[d.properties.InterZone][selectedColumn] })
    .attr("fill", function(d){
    const c = color(pricesMap[d.properties.InterZone][selectedColumn]);
    if (spend >= pricesMap[d.properties.InterZone][selectedColumn]) {
      return c;
    } else {
      return d3.color(c).copy({ opacity: 0.1 });
    }
  });
};

The first thing updateFill function does is read the user's budget input from the range slider (amountInput). It then calculates the corresponding monetary value (spend) based on this input, and updates amountDisplay label beside the slider.

The function then iterates over all the map areas (path elements) and updates their fill color based on the calculated spend. The 'budget' condition spend >= pricesMap[d.properties.InterZone][selectedColumn] basically determines whether or not opacity is applied to the region's fill colour.

This is what creates the 'blackening out' effect when you decrease the amount available using the slider.

Note, D3 has really neat functions for parsing and manipulating colour spaces. In this case, using d3.color to parse the colour value and apply opacity of 0.1.

3. Event handling

Since the user can change a number of things about the visualisation, there's a bit of event handling going.

0:00

amountInput.on("input", updateFill);

updateFill is called in two places, i.e. whenever the user interacts with the amountInput slider, and at the end of the render function itself.

render itself gets called once at the end of the script block, so the visualisation shows when the page is first loaded. It is also called when the user selects a different radio button.

const radioButtons = d3.selectAll("input[name='column']");
radioButtons.on("change", render);

As you'll have noticed, there's quite a lot going on in render, but it needs to be that way since we have different min and max values, so therefore the colour scale needs to change in line with the selected columns.

Advanced zoom interaction

You may be wondering what's going on in the zoom handler code as its a bit more extensive than the starter example in Part 1.

When you zoom and pan the map, the visualisation highlights the highest and lowest sale price data on the map with a little circle and label.

There's a lot to it so I'm going to write a dedicated post on how this bit works – especially as its one of the more challenging features of a visualisation I've worked on!

Choropleth mapping house sale prices - Part 1. Data Prep

Rory Gianni — Fri, 03 May 2024 14:12:48 GMT

🗺️

If you're looking for the dataviz, here it is.

Looking at the stats, the 'Choropleth code walkthrough' has been the most popular post/video by far. On that note, I thought I'd do another Choropleth map. This time I've sprinkled in interactivity to demonstrate d3's 'update' behaviour. In this post, I'll outline the rationale for the visualisation and how to prepare the data for it.

The Visualisation

Sometime soon I'd like to move house. If you've ever been in this position, the market somewhat comes to dominate your thoughts: where can I buy? how much can I sell for? etc.

All sales (where I live) in Scotland are recorded by the Registers of Scotland (RoS) and that data is published openly on the government's stats portal.

Those figures are made available for various boundaries including what's called '2011 Intermediate Data Zones' – for a fairly high resolution choropleth map. Here below is the end result:

0:00

A video of the visualisation. Alternatively, you can try the real thing here.

The slider controls allows you to choose an amount available towards a house. Areas above or below the value selected are dimmed or activated depending whether you go over a lower quintile, median, or mean value (also controllable).

Getting Price Data

The map reflects data collected for 2021 and 2022. This is the most up-to-date data that was published by the RoS. Other years are available all the way back to 2004 (in this dastaset anyway).

Although the dataset goes to 2022 there are inflation statistics that can help the prices reflect a more recent value. For this I used the UK House Price Index's monthly percentage change for Scotland from Jan 2023 to Feb 2024.

The pricing dataset exported from the stats hub was in great condition so I didn't have any cleaning to do. The only thing was figuring out how to actually export the data I wanted using the 'data cart' feature. The thing that threw me off was needing to navigate away from the dataset to the 'Atlas' in order to add desired rows to the export. It's not immediately intuitive so you need to read the guide rather than jumping in as I usually like to do.

Here's a video of the process that might be help:

0:00

Building a 'Data Cart'

Preparing the Boundary Data

The Scottish stats hub has a companion spatial data hub - spatialdata.gov.scot where you can download various open boundary data. In this case, the 'Intermediate Zone Boundaries 2011'.

The boundaries come as ESRI Shapefiles, so they need some conversion before we can use them with the d3-geo functions. For this purpose, MapShaper is a tremendous tool that does what we need.

💡

I owe a lot to this tutorial by Dylan Moriarty that helped me get a start with MapShaper.

At first I used the MapShaper GUI to open the Shapefiles, and before converting to TopoJSON simplify the map boundaries somewhat.

0:00

Using the 'Simplify' feature in MapShaper

Troubleshooting a broken map export

When I first exported to TopoJSON and attempted to render the map with D3 I saw nothing. Fiddling with the projection parameters I was able to produce a big mess. Inspecting the boundary paths I could see the 'd' attribute had 'NaN' scattered throughout what should be a string of coordinates. If you know your JS, NaN = "Not a number".

The step I left out was to specify the coordinate system. On this topic, I can recommend this tutorial by Flourish. The fix was fairly simple. In Mapshaper you can convert CRS (Coordinate Reference System) using -proj.

Referring back to the boundary metadata show CRS (EPSG:4258), so the command I needed to run was:

-proj from=EPSG:4258 crs=EPSG:4326

NB- Once you do this you'll see the map looks a little squashed, don't worry that's meant to happen!

Rendering the TopoJSON in D3

Naturally our start point is just visualisng the map data. I've put the code for this on Glitch.

The above map is rendered in 22 lines of code, though most of the magic happens in just a few of these.

Lines 7-11: are just about sizing the SVG to fill most of the window minus the size of the header.

Lines 14-17: After that, we grab the TopoJSON file we generated using mapshaper. d3-geo uses GeoJSON, but since TopoJSON is a much more compressed format you'll frequently see people use TopoJSON with the topojson-client library to make the final conversion on the client side.

After we have GeoJSON, called geoFeatures here. We can configure our d3 projection with the GeoJSON data plus details about how we want to render the map - i.e. how we want to convert coordinates into x, y pixels that will make sense for SVG elements.

For this purpose fitExtent helps configure the projection by setting dimensions and centring the projection on the centre-point of our boundary data rather than the default coordinates (0°N 0°E ).

Lines 20-31: Once we have a projection function defined, we use this create our pathDefiner function. This is the geographic path generator, the function that will take coordinates and turn them into SVG Path parameters.

The data zone boundaries are collected in feautres as an array. This is what we use for our data binding, so for each GeoJSON feature we append a path element. The pathDefiner is then used to set the "d" attribute which describes the shape of the path.

Zoom and Pan behaviour

Beyond just displaying the map, we have some zoom and pan behaviour.

D3 helps you define this behaviour with d3-zoom module. Basically it helps you work out how to transform SVG elements when you drag or scroll-in / out. That's what happens here, where I select all path elements in the areasGroup, and updating the "transform" attribute.

You'll see how this works if you open the element inspector and move the map around.

You might be wondering what the skip % 3 condition is doing here. This isn't strictly necessary. It's a way of debouncing the 'zoom' event as I noticed the zoom behaviour was a little slow on mobile. Simplifying the map boundaries would be another way of dealing with this issue.

Looking ahead

If you've inspected some of these path elements you'll have noticed ids for each of these boundaries. These match up with the boundary ids we have in the sale prices spreadsheet. In part II I'll cover matching these up so we can shade each boundary using d3.scaleSequential and d3.interpolateCool.

On that note, you can always view or remix the full visualisation's code on Glitch.

Building a Stripe data export app with Bubble

Rory Gianni — Thu, 04 Apr 2024 16:42:59 GMT

Over the past few weeks I've been dabbling with Nocode tool Bubble.io. I've played with Make, Airtable, and IFTTT before, but nothing as involved as Bubble. As a developer, Nocode holds two attractions, the promise of accelerated development, and second, the opportunity to empower non-technical colleagues.

Introducing Timely Exports

To learn, I decided to set myself a project - an app to periodically email an export of data, 'Timely Exports'. Recently, I've been working with the Stripe API a bit so I went with Stripe as the platform. The ideal workflow for this app was: user signs up to the app, chooses what resource they'd like to export (e.g. Customers, Checkout Sessions etc) and then choose how frequently they'd like to receive email exports. Done.

An example response from the Stripe Customers API endpoint

With Bubble, you more or less immediately have an application deployed with user accounts and email integration. The User account including login / sign-up forms are available right from the start. This means that save for UI details/branding, you can jump into implementing business logic straight away.

v0 Implementation Plan

All this application really needs to know is a little about the Stripe account in question (i.e. a name plus API key), and the details of what to export (resource name and periodicity).

But before jumping into setting up recurring events or allowing the user to select a resource, I needed to check I could achieve the following:

Allow the user to provide their (restricted) API key
Successfully call the Stripe API
Flatten the JSON response into a tabular CSV string
Save the string to a file
Attach said file to an email and send it

Making Stripe API requests with Bubble

There are lots of Stripe plugins for Bubble with API actions ready to go. However, I quickly found that few support all the capabilities of the API. This meant I needed to hand roll an API action to retrieve say, a list of customers. The way this is done in Bubble is you use the API connector plugin and configure the endpoints using a form.

This seemed simple enough. Fill in the form for each API request using the docs Stripe API docs for reference.

Retrieve a list of customers

In Bubble, the form is setup so that the authentication method is shared across all endpoints with credentials saved in advance so you don't provide them when it comes to invoking the API action.

Stripe requests are authenticated by using HTTP Basic Auth and using the secret key as the username, and leaving the password blank (no idea why they've gone down that route).

Hitting the Nocode wall

The above is sensible enough, but not at all what I wanted!

The token should be provided when we invoke the API action - just like any other parameter (e.g. 'limit' in this case). Unfortunately, it doesn't look like this plugin allows for passing auth details when you invoke API actions.

However, the API builder plugin allows you to pass in request header information. Among other things, this is what most of the various authentication methods actually do. The HTTP Basic Auth method Base64 encodes the username and password so you end up with an Authorization header that's something like this:

Encode "this_is_your_api_secret:in_stripe_this_blank" on base64encode.org and you'll see where the above string comes from

I was almost away to see if there was a way to Base64 encode until I remembered a simpler way to use basic auth. Prepending the user:pass combo with '@' in front of the domain:

Using square brackets here means Bubble understands that part as a parameter. Problem solved!

Using the initialize call button I was able to see that I could succesfully call the Stripe API. It seems Bubble requires you to do this so it can anticipate the contents of the API response.

Flattening the JSON response

The next step was translating JSON into a 'flattened' tabular format that would make sense as a CSV.

I found a plugin for this, but for whatever reason I couldn't get it to work. I flailed around trying to see if I could cobble together a few Bubble actions to achieve building a CSV string but that turned out to be pretty onerous. Also, it was fixed to the 'Customer' API response rather than being abstract. At this point I decided to see if I could inject some code via my own plugin.

I thought this was going to be a huge hassle, but what's nice about plugins in bubble is that many plugins are openly readable and you can see exactly how they're written.

Using a few examples helped me jump into writing a JSON to CSV string function in JS that I could invoke in an action workflow where the input would be the raw response (as text), parse it, then flatten it into a CSV using the JSON keys as headers. For laziness/speed reasons I decided to ignore nested keys.

Aside from the raw body text, this function can take in a path specifier. This was to provide a bit of flexibility so you can tell the function where the collection is to turn into a CSV. For this purpose I require the 'jsonpath' package that helps you navigate JSON objects using a path string.

There's one other parameter this plugin action takes, and that's 'encode_output'. This is an option to Base64 encode the output which comes into play for file uploads in Bubble.

What happened to the nocode?

I know. I know. There probably was a Nocode option out there for achieving this. From what I've gathered, full on nocoders would probably turn to looping in outside services like Make and Zapier in order to achieve bits of logic that aren't available in Bubble. I get the impression that any one nocode tool can only get you 80% of the way before you hit a wall.

Anyway, I digress.

File Uploads

The next step was to take the result of this function, a big long string, and upload it as a file. Bubble has a 'Upload as CSV' action, but it demands that you specify an object / collection of objects from your db to upload. Once again I needed to figure out something myself. This is where the forums were very helpful. I discovered a thread that explained Bubble's file upload API. Using this I was able to construct an API action to upload to Bubble's file storage.

Bubble's file upload API expects 'contents' text to be Base64 encoded so that's why I added that option in the JSON to CSV action.

This is what the workflow looks like, chaining together the results of the various steps up to the API call action.

Sending Email

Sending email using Bubble is dead easy! There's an inbuilt action. The bit that wasn't immediately clear was how to attach the newly uploaded file.

It turns out the response body of the upload API request has the URL to the file. This is what you need to provide in order to attach a file in the email action. Simple enough.

Here's the workflow and UI ended up with.

I added a Test button that doesn't do the upload step, but rather just displays the unencoded CSV text.

Since I don't want to end up storing anyone's data, I added a final workflow step to delete the uploaded file.

The End Result

A spreadsheet, in my email inbox. Much Wow. These are the headings you get out of it. There'd be a few more if I bothered to sort out the nested object keys:

id  object  address balance created currency  default_currency  default_source  delinquent  description discount  email invoice_prefix  invoice_settings  livemode  metadata  name  phone preferred_locales shipping  tax_exempt  test_clock

An example customer export spreadsheet

If you'd like to try this app for yourself head over to https://post-emu.bubbleapps.io/version-test

Next Steps

In order to fulfil the original project vision, I'll need to provide a few more Stripe resource options for export (e.g. Checkout Sessions, Events etc.), and then persist the user's 'Export Configurations' to the database.

The workflow itself remains more or less the same, but I belive it becomes what Bubble calls a backend workflow in order to be invoked as a recurring event. I know Bubble has provision for recurring events so I feel like the above isn't far off. What I don't know, however, is how to securely store my users' API keys. A cursory look through the plugins does reveal a few encryption options though so it seems doable.

Some thoughts on Nocode

While playing around with Bubble on this project and a few other 'hello worlds' I hit several walls. This, in some senses, isn't dissimilar to my experiences learning new (code-based) frameworks. Each framework has assumptions built-in for ease and to accelerate development. The flip-side of this is that when they don't match these backfire and slow you down.

Like any technology, the trick is to know what you can and cannot easily do in order to be productive.

The making of Super Map World - Part 2. Core Architecture

Rory Gianni — Mon, 01 Apr 2024 15:01:22 GMT

This is the 2nd post in a series about building Super Map World (SMW). SMW is a web-application featuring over 17,000 maps and map graphics. They're free and they're customisable using a map edit feature. Head over here for the first post of the series.

To spend as much time focused on the core of the application I needed a simple template that would cut out the fiddling with project setup. My focus was on figuring out how to best represent and store the generatively designed maps from my prototype.

I wanted something that would get me started with user accounts, email integration, and all the other bits and pieces that make web-applications work. My two main requirements were, 1. nothing over-engineered, and 2. nothing relying on libraries that I wasn't somewhat familiar with.

A Quickstart with Speedrail

Using Github search for Rails templates I eventually came across Speedrail. Speedrail was made by Ryan Kulp for his 'Founder/Hacker' coding and product-launch course. It pretty much ticked all the boxes for me in my search for a simple, modern, Rails 7 template featuring:

Devise for authentication
Tailwind CSS
Stripe for payments
Rspec for testing
Postmark for email
Importmap for managing libraries

Getting started with Speedrail was a matter of cloning the repo, running the rename command bin/speedrail new_app_name, configuring environment variables in config/application.yml, then using it as you would any Rails application.

Core architecture

The most important part of any web-application is its data architecture. The question now was how to store the various elements I had in my prototype, namely, map styles, and the generatively designed maps themselves.

Map of Switzerland showing district boundaries

I needed something that could capture all the possible variations in a map rendering. Properties of a map can include several things including:

geographic data i.e. the file it was rendered from, resolution, recommended projection
source metadata e.g. where data was sourced, licencing details
projection i.e. is it mercator, orthographic, something else?
theme i.e. colours, shadows, graticule yes/no, borders yes/no, etc.
location details e.g. centre point coordinates, name of the location
visual adjustments e.g. margins and offset

It took me a while to come up with a schema that could represent all this while being 1. normalised, and efficent in database terms, and 2. easily indexed and searchable.

I came up with a 4 part system for capturing maps:

MapStyle – represents all the styling information that could be applied to rendering a map.
MapConfig – represents a source of geographic data and its relevent metadata.
MapFraming – represents how the geographic data is framed and projected. This means projection plus any adjustments such as rotation, scaling, or margins.
MapRendering – represents the rendering or a final image generated from all these factors. It has a number, a slug, and ties together these above representations via associations.

ER diagram of how they fit together

While the above architecture seems clear now, I only thought about having MapStyle, MapConfig, and MapRendering. I couldn't figure out whether the framing information should sit with MapStyle or with MapRendering. I passed it back and forth quite a bit until I realised I was dealing with something that deserved a life of its own.

Essentially, a MapFraming tells us how to use the geographic data to depict something then MapStyle provides the theme.

To illustrate the relationships between these different models, let's consider the above 3 images or 'renderings'. They all share the same data, represented by the MapConfig. The two maps on the right share the same MapFraming, they are both orthographic projections, rotated to be centred on India. They don't share the same MapStyle, unlike the two on the left that do.

Representing the above in the database there's 1 MapConfig, 2 MapFramings, 2 MapStyles, and 3 MapRenderings

In the Super Map World database there are around 450 MapFramings, and roughly 87 different styles. The combination of the two means (450 x 87) we can generate 39150 different renderings.

Locations and searching

You may have noticed in the ER diagram that MapFraming has a location_id. Location is associated with MapFraming. Its a model I use to support the search feature. The location table has been populated with countries and their ISO codes.

That said, the model isn't restricted to countries. It's intended to support any location on the globe that could be depicted with a map. Technically, a river or a city or a mountain could be the focul point of a map on SMW.

Currently, I have 'The World' as an entry in the location table because a world Mercator map isn't really depicting anything other than the world itself.

This location name information helps generate slug and description of map renderings. If you go to a map rendering's page on SMW, you'll notice this used in the description field along with the projection, boundary details (if any), and colour choice.

Reasonable Colors

Reading through the description field you might wonder where the colour names come from. In the previous post, I explained the style generator randomly selected from a list of CSS colour names in order to come up with a theme. If you know your CSS names you'll notice these names don't exist in the CSS colour set.

After my initial prototype, I decided to ditch CSS colour names as they aren't quite uniform in their distribution. Thankfully, I came across 'Reasonable Colors' by Matthew Howell.

What's neat about this project is its simplicty and approach to accessibility and contrast.

because this is all built within the LCH color space and the relative luminance for each shade is pinned within certain ranges, those contrast rules work across all 24 color sets. Mix and match shades from any color, even the grays.

Each colour comes with 6 pre-defined shades. For SMW, it provided a comprehensive yet manageable set of colours to choose from.

Constraining to this set also meant it wasn't too onerous to come up with a few extra colour names to make the descriptions a little more interesting. More importantly, it meant I could come up with a search feature that wouldn't demand the user to put in hex or rgb codes.

When you click on a colour on the search, or on a map rendering page it'll use the colour name plus shade index to query map styles and associated map renderings.

Colour search - note the ?colour=amber-2 in the query string

JSONB fields

Looking at the ER diagram you might also be wondering where exactly all this theme information is being stored given MapStyle only has 6 fields. Answer: it's all stuffed in the 'body' field.

That might sound horrible but I quite like it for this case. You see SMW uses PostgreSQL, and PostgreSQL supports what's called a 'JSONB' field where you can store arbitrary JSON objects. In my previous SMW post, I described my approach to generating map styles and the schema I came up with. Not wanting to alter or flatten this into a collection of relational DB tables, I opted to keep it as is, in a JSONB field.

What's neat about this is its flexible and we can still use Postgres to query values in there. I do use a custom Rails validator to ensure nothing funny goes on.

Here's how I query map renderings for a given colour or colour shade combination in map_rendering.rb:

When a map style is generated I include a list of the colours and shades used in the style. This makes for handy reference and indexing. I realise it might be worth extracting this information out into a field of its own for faster indexing. For the moment, it seems to work fine.

Next up: data and rendering

Aside from map styles I haven't quite explained where I got the geographic data from, and how that goes on a journey to becoming the map renderings you see on SMW. If that post doesn't get too extensive I'll cover a bit of the Stimulus and d3 JavaScript that brings things together.

Btw, if you haven't already, do play around with SMW! Feedback is appreciated! You can also find SMW on Instagram.

I will be extra grateful if you could share with your carto, map, or graphic-design inclined friends. This can contribute to the post on SEO / marketing (even though I don't have much to teach in that regard).

The making of Super Map World - Part 1.

Rory Gianni — Wed, 21 Feb 2024 13:49:55 GMT

Super Map World is a site that offers over 17,000 maps and map graphics. They're colourful. They're free! And best of all, they're customisable using a map edit feature.

It's one of the biggest side projects I've committed to. Over the course of the next few posts I'm going to write up my approach to, and experience of, building Super Map World (SMW).

I'll cover:

The tech stack (Ruby on Rails, StimulusJS, D3js, Tailwind)
Gathering geographic data for 200+ countries
Deployment and configuring a CDN
Challenges and how they were overcome (or worked-around)
Lessons on launching SMW onto the web
Developing an add-on for Adobe Express

0:00

Version 0

My favourite types of data-viz / creative tech projects for Creating with Data have involved generative design and maps. I began to wonder if there was some scope to combine these elements.

Taking a generative design approach is a great way to turn out effective and aesthetically pleasing graphics. Why not try it with maps?

I began a playing around with d3.js, rendering world maps in different styles.

0:00

This version randomly selected colours for the land, water, and (optionally) graticule from a list of CSS named colours. Graticule, shadow, and stroke widths were randomly decided too.

This was the job of a generateStyle function, that output a mapStyle object containing the various visual properties to be applied at time of render. Here's a little snippet from the function:

{
  "land": {
    "hasShadow": false,
    "visible": true,
    "fillStyle": randomColours[0],
    "strokeVisible": (Math.random() >= 0.5),
    "strokeStyle": randomColours[1],
    "lineWidth": 2 * Math.random() + 0.05
  },
  "water": {
    "fillStyle": randomColours[2],
    "strokeStyle": randomColours[3],
    "strokeVisible": (Math.random() >= 0.5),
    "lineWidth": 2 * Math.random() + 0.95
  },
  "graticule": {
    "visible": hasGraticule,
    "strokeStyle": randomColours[4],
    "lineWidth": 2*Math.random() + 0.05
  }
 ...

Land, water, graticule, are all elements of any map you might generate. Later on, I added "borders" to the object to control the style of boundaries. Most of the keys here correspond more or less to the names of the functions call when drawing with the Web Canvas API.

Contrast correction

At first, random selection meant some colours were the same or too close. Landmasses and lines ended up barely visible in the odd rendering. The theme generator needed to correct for contrast so I ended up with a function that reselected until contrast was high enough.

// Function to randomly select pairs of colors until the contrast value is above min

function findColorsWithContrast(color1, requiredContrast) {
  if (requiredContrast === undefined) { requiredContrast = 1.22; }
  let color2, contrast;
  let i = 0;

  do {
    if (color1 === undefined) { color1 = getRandomColorName(); }
    color2 = getRandomColorName();
    contrast = calculateContrast(color1, color2);
    contrast = Math.round(contrast * 100) / 100;
    //console.debug(`%c■■■■ ${i} ${contrast} ${color1} ${color2}`, `color: ${color1}; background: ${color2}; font-size: 20px;`);
    i+=1;
  } while (contrast <= requiredContrast);

  return {
    color1: color1,
    color2: color2,
    contrast: contrast
  };
}

Since colours in CSS can be defined in a number of different ways (e.g. rgb, hsla, 'black', etc.) d3's 'd3-color' module was dead handy. It helps you parse those different colour spaces and manipulate them. Digging into colour spaces was an eye opener.

Do you know your colours?

This is a bit of a tangent, but one worth mentioning. For most of my career I've defaulted to RGB and HEX but this experience helped me understand the benefits of using HSLA colour expressions. It also introduced me to the LCH colour space. Lea Verou's article on the topic is incredible -

"Today, the gamut (range of possible colors displayed) of most monitors is closer to P3, which has a 50% larger volume than sRGB. CSS right now cannot access these colors at all. Let me repeat: We have no access to one third of the colors in most modern monitors."

Verou on LCH. I am still getting my head round how this works!

The vision appears

When I set out, the intention was to come up with an interesting example to post here on Creating with Data.

Rather, what happened was that I was seduced by all the possibilities and combinations that could be realised with the same approach - randomising all the different projections, styling boundaries, highlighting regions, using transparency, using geographic data from different sources, so many possibilities!

A thassalaphilia kicked in. I wanted to dive in deeper.

Extending the features of the single page experiment, I began to imagine a web-application that could be used to quickly create attractive maps and map graphics. Something in the spirit of colour palette inspiration tools such as Coolors or ColorHunt, but also with the utility of a easy infographic and dataviz tools such as Datawrapper or RAWGraphs.

Clockwise from top left, RAWGraphs, DataWrapper, Coolors, ColorHunt

The start point was a single html file plus a couple of JS files. The goal was an application that could support users and offer them value added premium features – all while supporting open access.

🎵 I hear that train a-comin'

As this was becoming a bit more ambitious. I decided to reach for the web-framework that I know and love best, Ruby on Rails. It's arguable a JS-based framework would have made more sense considering I was already knee deep in Javascript. That said, I know first hand that employing shiny new tools is a deadly distraction for any new project.

Getting this up quickly was also a priority. Learning a new framework was not on the critical path. And, as much as I get a rush typing `rails new`, I decided the smarter move was to start with an open source template app.

In the next post, I'll go through the first steps in setting up the Rails app that became Super Map World.

Btw, if you haven't already, do play around with SMW! Feedback is appreciated! You can also find SMW on Instagram.

Take your first steps with Data Driven Documents (webinar recording)

Rory Gianni — Thu, 05 Oct 2023 13:45:08 GMT

Last week I ran a webinar for The Data Lab Community. The aim of the webinar was to provide a hands-on introduction to D3, and cover:

What D3 is, and what it's not
Capabilities of the library
How to use 'data-bindings' to power your visualisations
How to manipulate SVG elements to represent your data
Where to go next

You can find the tutorial materials over here if you want to follow along.

Code & Learn: Set your mind in motion with a project vision

Rory Gianni — Mon, 21 Aug 2023 14:22:53 GMT

Figuring out where to start with a new programming language or framework can be challenging. Searching for advice can inundate you with long lists of prerequisites and concepts to learn. Forums will turn up tech veterans that demand newcomers "start from the ground up".

The newcomers should know the fundamentals before they even begin to touch that framework or library. For them to do so would be to meddle with things beyond their comprehension. Heavens! It could lead to their breaking the fabric of the Internet. Or something.

I'm sorry to say I opened Pandora's box and it worked out just fine.

I didn't know Ruby before I started learning Ruby on Rails. I learnt the two in tandem. Sometimes it was confusing what was Ruby, and, what was Rails. I had a decent grasp of desktop software development, but not web application development. I didn't have a clear knowledge of REST architecture.

Regardless, I ended up proficient in Rails because I took on personal projects, and was keen to see my ideas realised.

Start with a vision

"Come up with a project idea you'll enjoy and try to build it" has been a mainstay of advice to anyone I've met learning to code over the years.

There's a terrific post from Ian Johnson called 'How do you learn d3.js?'. In it he asks several well-regarded information and visualisation designers about their respective journies. All of them mention a vision of telling a story with data or creating something that didn't exist before.

💬

...everything I was learning was driven by “I want to do X… now how do I do that?” - Zan Armstrong

It's reassuring to know this approach works for others and not just me.

"My vision, one nation, one tribe. One day'll come the might to move any mountain. Move any mountain." - The Shamen (Move any Mountain Lyrics)

Why projects?

Getting excited and remaining engaged is one of the main reasons I advocate taking on a personal project. That motivation will provide the fuel for you to continue.

Motivation isn't the only reason. When you try to realise an idea you'll be required to overcome equally unique challenges. There may be examples and libraries to help, but you'll need to adapt any prior content to the context of your project. This process of creative synthesis is of a higher order of thinking and problem solving.

You might find it ambitious to set to work on a passion project on day 1 of learning a new technology. Still, it's definitely worth thinking about. Contemplating your project vision is a way to set your mind in motion.

Carrying that vision with you while you're learning will predispose you to notice important details that crop up in lessons or in your own reading. You can think of this as a net, deployed subsconsciously to gather valuable nuggets of insight.

The net that is cast. This is the Cingulate Cortex as drawn by Santiago Ramón y Cajal. This part of the brain is responsible for among many things, learning and memory.

Constraints can guide

Building a project may sound unstructured compared to following a set syllabus or roadmap, however, choosing a project provides constraints. Constraints are your friend because learning any technology can be overwhelming. There are heaps of resources on the web. This should make things easier but unfortunately it can make life hard when you're unsure where to start.

If you're able to break down the component requirements of your project, you'll have a roadmap to guide you.

Let's say your aim is to create your first data visualisation using d3.js. It doesn't involve any interactivity or fetches of remote data. That's fine. You can dispense with chapters on event handling and asynchronous programming. You've just saved yourself a lot of energy and time!

Caveats for project based learning

I believe those are the strongest reasons to value a project-based approach to learning. There are some other benefits too such as building up a portfolio of projects for demonstration and providing a sense of achievement. That said, I'd like to add a few caveats.

Caveat 1: Don't bite off more than you can chew
This sounds a fairly obvious point to make. Taking on something too complex will lead to frustration. You knew that part. The thing is, experienced developers occasionally underestimate how complex projects or features may be. You will do the same. I recommend choosing projects that you can add to and enhance over time.

Caveat 2: Recognising when to step back
At some point working on your project you'll get stuck. Most of the time (hopefully) you'll be able to debug or Stackoverflow your way out of it.

There may come a point where you're really stuck. Not only that, you're not sure what to search for, or even how to articulate a request for help. This is a sign you need to learn more about an underlying technology or a fundamental concept. Step away from the project and pick up a book.

Alternatively, this could also be a sign that you're not getting all the information you need to diagnose the problem. You may need to learn more about the ways you can make use of your IDE or developer-tools to set breakpoints, read, or output logs.

D3.js Spinning Globe Code Walkthrough

Rory Gianni — Thu, 27 Jul 2023 12:08:40 GMT

In this walkthrough, we look at the code behind this spinning globe graphic rendered to web canvas using D3.js.

Links:

Using a text input to filter map data with Leaflet JS

Rory Gianni — Thu, 20 Jul 2023 09:00:00 GMT

In this follow-up video, we add another method of filtering data on our doctors' practices map visualisation.

If you're not too familiar with Leaflet or JavaScript in general, I recommend watching the first Leaflet.js video where we create our visualisation from scratch.

You can find the complete map visualisation here and code for the start and end point on Glitch.

Dataviz: Connecting the Atlantic

Rory Gianni — Wed, 12 Jul 2023 18:09:41 GMT

“A treaty of peace has been signed between Austria and Prussia” – the first message that was transmitted across the Atlantic via telegraph. That's the first connection that appears in this animated globe visualisation. It shows a timeline of various trans-Atlantic subsea telegraph connections starting with the first successfully laid cable in 1866.

0:00

Background

I've been looking out for something to combine globes / geospatial data and animation with d3 for a tutorial video. Listening to the audio book of Andrew Carnegie's autobiography, he describes one of his first job as a messenger boy for a telegraph firm, along with interesting anecdotes about the infrastructure of the time (mid-19th Century). This gave me the idea to look into the history a little more and I discovered some excellent resources. The highlight was this site jam-packed with timelines and history on undersea communication infrastructure. You have to love the classic table element web-design.

They host some lovely maps too!

Geocoding and converting to JSON with ChatGPT

Helpfully the timeline information was already in a table which I saved to CSV format. It didn't have coordinate data for the routes and I knew I'd rather work in JSON. I thought I'd take a shot at re-structuring the table into JSON and asking for coordinates for the start and end points of the routes. To make things simpler I split up certain routes so they had a single start and end point. I haven't double checked every coordinate provided by ChatGPT but they've been accurate enough for the purposes so far!

Mapping and filtering prescriptions data with LeafletJS (in-depth)

Rory Gianni — Tue, 11 Jul 2023 10:54:00 GMT

In this video, we visualise data from a spreadsheet of doctors' practices and prescriptions adding two methods for filtering. We deal with the issue of combining the two separate filters as one has a knock-on effect on the other.

NB- the data is synthetic, it's just modelled on the real thing!

This video is on the long side. If you're more interested in the visualisation than the control/filtering stuff you can just watch the first 20-25 mins!

Notes

You can find the starter template and the finished article on Glitch.

Edit: Make sure to check out the follow-up video!

D3.js Choropleth Map Code Walkthrough

Rory Gianni — Mon, 03 Jul 2023 09:14:00 GMT

Maps are some of the most shared and sought after forms of visualisation. In this video, I walk through an example that visualises unemployment data at the US county level. The code was updated from this bl.ocks.org v4 example to use d3.js version 7. The walkthrough touches on using d3-fetch to retrieve data on the web, as well as some basics of rendering geographic data.

Notes:

Choropleth map code on Glitch
'Map' object documentation on MDN
Topojson repo on GitHub
'Introducing asynchronous JavaScript' article on MDN

Creating data-driven fabric designs with JavaScript - P2. coding the generative designer

Rory Gianni — Mon, 19 Jun 2023 14:43:15 GMT

This screencast shows how to code a generate fabric patterns and visualise them as shirts using SVG and D3js. The patterns are composed of images gathered from an archive of botanical watercolours held by the US Department of Agriculture.

For some context about the project I've written a post about how I used an early version of the designer to get the fabric printed and then tailored.

As the title indicates, this is part 2, if you'd like to know more about how the images were gathered and prepared from the watercolour archive with NodeJS, check-out part 1.

Here's the screencast, and here's a link to the starter code on Glitch.