The data behind data-led design

I have been making an experiment about turning data into useful creative work. Over on AceMedia Ninja, Data has a pulse takes the visual side: campaign materials, maps, charts and a small world with a floating jewel. The Data Pulse Lab lets you change the inputs yourself.

This is the other half of that work. A good-looking map can still contain yesterday’s opening hours. A fast page can still show a price whose source has stopped updating. A forecast can look impressive because someone chose the week on which it happened to work.

I want the design to make the evidence easier to understand. That starts with deciding what each field means, where it came from and what the system should do when it cannot trust it.

A record needs more than a value

Take a council activity listing. The useful fields might include a venue, a start time, cost, accessibility information and places remaining. Each field answers a different question. “Step-free access confirmed” is different from an empty accessibility field, and neither should quietly become “fully accessible”.

I would keep a stable record ID, a named source, the time the source says the information was measured, the time our system collected it and a clear unit. The source time and the collection time are separate. Fetching an old record again does not make its contents new.

Validation comes before the drawing code. A negative capacity, an unrecognised unit or a timestamp in the future needs a visible decision. Sometimes the right decision is to withhold a claim and link to the organiser. There is no benefit in making bad information travel faster.

Keep source health separate from content freshness

Work with changing publishing feeds has made this distinction particularly useful. A source can be unavailable while the last saved record is still useful. It can also return a successful response containing old information. I want to see both the result of the request and the age of the content.

This fictional listing has twelve places available. Its display rule allows an availability claim for ten minutes after the source observation. That is a made-up rule for this example, not a recommended lifetime for every service. Change the source response and age to see what the customer would be told.

{"places_remaining":12,"source_observed_at":"12:00","captured_at":"12:00:30","evaluated_at":"12:02","claim_limit_minutes":10,"latest_request":"ok"}

The same principle applies to a shop’s stock display, a venue listing or an infrastructure summary. A failed request is a problem with that request. It is not proof that a venue has closed, a product is sold out or a service has stopped.

Useful history records transitions and their time windows. A timeline showing “answered”, “unreachable” and “recovered” can help an editor understand why an update paused. Where checks run periodically, I also preserve the sampling interval: a fault first noticed at 12:10 did not necessarily begin at precisely 12:10.

A cache needs a policy the customer can see

I prefer a bounded background refresh to asking every visitor to wait for an upstream service. The page can use a saved response while a scheduled job checks for a newer one. That job needs timeouts, a retry budget and a way to avoid several workers repeating the same expensive request.

But “we have a cache” is only half an answer. The other half is deciding what can be shown at each age. An old biography might remain useful. A stock count, transport disruption or price can need a much shorter claim window. A timestamp and a clear fallback let the interface remain useful without pretending every field is current.

I wrote more about the operational side in Background cache refresh needs a budget too and Show what your monitoring has actually checked.

Forecast attention, then measure the error

Historical counts can help plan content or capacity. They do not turn a click into a completed booking, or a market update into a prediction of a winning bet. Those are different outcomes, with different evidence.

For a forecast, I keep the order of time intact: make a prediction using information available at that point, then compare it with the later observation. A simple baseline matters. If a complicated model cannot improve on “use yesterday’s count”, I want to know before anyone plans a campaign around it.

In this fictional example, both forecast series are declared in advance. The first four days flatter the candidate model. The later days change the picture. Mean absolute error is the average distance between each forecast and the observed count, measured in the same units. Lower is better for this particular measure.

I would also look at the direction and size of individual errors, the periods where information was missing and whether the forecast helped someone make a better decision. Choosing a model after inspecting an evaluation period uses up some of that period’s independence. The next test needs later evidence.

Join measures without losing their boundaries

A campaign report might contain page views, tagged link clicks, enquiries and confirmed bookings. I keep the definition, time window and collection scope beside each measure. If the sessions and clicks were collected under different consent or filtering rules, dividing one by the other does not magically produce a trustworthy conversion rate.

Attribution has its own traps. The person named in an article’s byline may not be the person who updated its data or placed a campaign block. A useful content model can preserve that work without counting the same article’s engagement twice when it appears under two contributors.

For public examples I use invented aggregates. For an actual service, I would agree which question the team needs answered, which source can answer it and which people may see the result. Council service coverage, community access and individual behaviour should not be mixed into one apparently convenient dataset.

Shared operations, separate sites

The infrastructure can make those boundaries easier to maintain. A proposed managed estate can give each site its own content, credentials, data access and recovery path while sharing a consistent release process. A verified code update can move through a pilot, a set of checks and a staged rollout. If one site fails its checks, its rollback should not depend on undoing every other site.

That is the distinction behind the AceCore and AceLink work: eligible shared code is one concern; coordinating inventory, releases and per-site checks is another. Adoption still needs a fit review. A common dashboard is not permission to merge every organisation’s data.

The front end should get the same care. The examples here initialise as you approach them, do not poll a service and show their definitions as ordinary text. The larger visual experiments stay in the Data Pulse Lab, where they have room to explain their own controls.

I enjoy the moving jewel. I also want to know why it moved, how old its input was and what happens when the source disappears. That is what makes the creative worth keeping.

Let the infrastructure map explain the customer-facing claim

A map is useful when it makes a dependency inspectable. In my hosting control-plane work, that means separating configured relationships from observed activity and keeping the evidence beside a node. This public example uses fictional resources and a much smaller chain.

A booking source supplies a saved record. A media library supplies approved artwork. An origin and edge cache deliver the page, and the same record can populate an exported screen graphic. Select the availability line to trace the claim back to its record. Change cache share to see delivery change without pretending the offer or audience has changed.

The inspector keeps observation age, collection age and request status separate. A source outage can leave a recent saved observation usable; an expired observation cannot support a fresh availability claim. The exact limit is part of this fictional example, not a universal caching recommendation.

Use an editorial signal without inventing an audience

A useful campaign pipeline can collect permitted news or content metadata, group it by topic and compare equivalent windows. The output is a source-backed brief for an editor. A rise in articles about a subject is not automatically a rise in customer demand, and copied headlines should not become a marketing feed.

The study below uses invented, fixed-size samples. A chosen context changes the visual rule and a chosen voice selects authored copy. When the evidence is marked unverified, the claim and data marks are withdrawn. That is a useful boundary between a system preparing options and a person approving a campaign.

For a real service, I would define source permissions, retention, topic matching, duplicate handling, sample coverage and approval ownership before wiring it into production. Measurement then preserves the creative version and agreed collection scope. A campaign click is a click, not a confirmed purchase or a unique person.

Where this becomes a service

Data pipelines and publishing systems covers collection, record definitions, freshness and reviewable exports. Campaign infrastructure covers the landing pages, media delivery, launch checks and operation behind the creative. For the concept, illustration and motion itself, AceMedia Ninja develops the campaign.