Blog Title

Designing an Architecture for StreetSmart

2026-04-11T00:00:00Z

During a 2025 climate-tech fellowship through Climatebase, I led the technical side of StreetSmart, a digital platform that turns street images into usable insight for cities. Built in close collaboration with urban planner Reena Mahajan and a multidisciplinary team of designers and engineers, the platform makes everyday street experience legible, identifying what makes a street work, or not, for the people it's meant to serve.

A person uploads a picture of a street and our app shows parts of the street design that make it good or bad to walk around/live in. Email me if you want to join the private beta!

In what mostly amounted to a three-month sprint from product conception to demo day, I led a small global team of software engineers to design and build our AI-powered MVP in collaboration with urban planners. It was successful by most measures: we formalized urban planners' notions of a liveable street, made a working prototype, ran two pilots with partners in India, and are now attracting a fresh batch of volunteers for ongoing development.

I'm continuing on as an advisor, and wanted to write down three architectural decisions that shaped the codebase while they're still fresh:

Use tech that the team knows
Maintain a $0/yr annual budget
Don't blindly trust an LLM

Context: What we built

From the outset, our MVP was to build an app where people could take a picture of a street and the app would call out aspects of the street's design that make it good and bad for liveability/walkability. We chose to build this as more of a small, loveable, correct app rather than as the shell of an enterprise application with a lot of dead ends.

We worked with a team of urban planners and architects to articulate concretely what aspects of the street's design influence liveability/walkability. This team still has many quibbles with what our app recommends, but on the whole think it gets people to notice and think more carefully about their living environment.

The app is designed to work on desktop and mobile. Mobile because we assumed that's where most users would use it, desktop because we needed to test and build it on our computers.

Decision 1: Use tech that the team knows

As a volunteer group, we were limited to whatever people could be inspired to contribute. Which, between working other jobs/school/job hunting, was limited.

We settled on a doacracy mindset, where people who do work have a strong influence on how the work is done. In practice, I'd scope out a component — what it could do and how it might deploy — and a contributor would be responsible for vetting those assumptions and delivering something the project would find useful. I'd then integrate their work into the main app.

We decided that we'd make a web app. The team had a lot of AI-focused data people on it and most of their experience (and most AI tooling) were in Python, so we opted to use that as our backend rather than a more trendy Node.js/Typescript.

Overall, this structure worked reasonably well and left us with these architectural choices:

a backend FastAPI microservices architecture (which was new to me) because one prolific contributor wrote a working draft of it and coached the rest of the team through deploying and supporting it to production
vanilla JS on the front-end (not React) because I had extensive experience in vanilla JS, saw little benefit to the complexity of the big frameworks, and others who attempted to build on other stacks were unable to integrate with our existing tech assets
lots of business logic in vanilla JS because the author had the most experience in it. We appreciated the low-cost in-browser testing where I could set up some flags for the page, edit the file, and then see my results in milliseconds after reloading the page. Another urban planner also learned how to edit these JS business rules and could use the already-installed Chrome Developer Tools to see the results of their changes.

In retrospect this led to some unfortunate outcomes:

by using vanilla JS rather than React, we may have limited our pool of contributors. The gap between our stack and what many developers used created friction: some contributors with deep React experience found it hard to transfer their patterns to vanilla JS, and others hoping to build React experience weren't motivated by the work. This led to a contingent investing hours in planning a React rewrite that never materialized into code (if only we'd been able to channel that energy into more incremental experiments). Similarly, a graphic designer spent hours vibe-coding a beautiful version of the frontend in Loveable, but we were only able to include it after painstakingly reverse-engineering small parts because the Loveable stack didn't work with what we'd already built.
by not being good at deploying Python we failed to integrate a privacy-focused contribution that required a different sub-version of Python. If only I had realized virtual environments could come to our rescue.
having key parts of our business logic written in JS made it hard for contributors with only Python experience to understand what was happening and effectively contribute

Decision 2: Maintain a $0/yr annual budget

The AI startup ecosystem is full of free trial offers. But free trials have two relevant periods: the free period, and then the paid period.

I had an intuition that the transition to the paid period of too many products would introduce uncomfortable questions for the project. I had no doubts that the founder was bought in to the success of the project, but I knew she had real financial constraints. Could she indefinitely invest $1/mo in the project? Probably. $100/mo? Probably not without a clear path to revenue.

Should we have signed up for these trials, we would have also had a lot more management overhead: Whose credit card would be on the trial? Does it matter whether the payer is in Europe, India, or the US? Who would keep track of how much we're spending or on track to spend? How did we avoid unbounded spending like people sometimes experience with AWS? Who would be in charge of winding down the service at the end of the free trial, and who would decide that when? Lots of questions, so few great answers.

I also quietly expected that the end of major trial periods would prompt a reckoning for the project: if it wasn't making tangible progress, our product would be unceremoniously shut down and future teams would be forced to start with scratch (or, rather, a bunch of low-quality code and some sketchy instructions about how to run it... which is pretty close to scratch). Conversely, if we were on free-forever plans, the project could lie dormant for a while without it prompting these sorts of shut-down decisions. It was also advantageous for the team, because I suspected some of us might want to use it as a portfolio piece, and it's more engaging to have a working product than a bunch of screenshots.

If we could keep it to $0/yr, lots of problems went away.

How we did it

Indeed, we managed to keep our infrastructure costs very low! Our only ongoing cost is $12/year for the internet domain, which I think is a pretty unavoidable expense. It's also one that makes sense as long as the project lives as a twinkle in the founder's eye, and can survive through several technical architecture changes.

We found acceptable free tiers of products:

Cloudflare Pages hosts the front-end code. It does it on a CDN close to the user, allowing fast page loads even if our API is a bit slow.
a free ARM VPS on Oracle Cloud hosts the Python backend, a local mask2former-based image segmenter, a PostgreSQL instance and stores our user-submitted photos
Google Gemini offered a few hundred calls per day for free, powering much of our more complex logic. Informal suggesting found that Gemini offered similar results to Claude or ChatGPT
GitHub's free tier handles a lot of our CI and collaboration

Sticking to the free tier gave us a few constraints:

Probably staying away from AWS was a good choice, although we may have been able to attract someone who had more experience administering AWS than I did with running a bare Linux server.
One downside of the free Gemini was our requests would fail when the Gemini infrastructure was overloaded. And unfortunately it often seemed overloaded while the technical Americans were awake and the urban planning Europeans were awake. Due to mediocre error reporting, this lead non-technical team members to thinking that something was wrong with our app when we could do little about it other than paying more. It eventually got bad enough that a tech person decided to fix it temporarily by paying for Gemini (with some strict cost controls) to keep the app working for our pilot events. Going forward we may re-investigate Gemini's free tier or look around at other models.
it mostly closed the door to training a custom model to recognize different parts of street design. Probably this kept us focused on articulating what we want, but it's possible that some model tuning could have gotten us better results.

Decision 3: Don't blindly trust the LLM

There was a stage of development where some of the tech team thought we could make the app work with just an LLM prompt with the response piped back to the user. We never tried this because of three concerns:

the urban planners (in particular) worried that blind trust in an LLM would reproduce/amplify existing biases in how streets are understood
I assumed that it would get us to a middling quality level, but then we would have things we wanted to change and we wouldn't have a ton of control over the system and as a result wouldn't have a great way to influence the outcomes.
mid-2025 LLMs weren't consistently great at making internally consistent outputs, so some application code helps ensure that our final analysis was consistent and avoids embarrassing outcomes from a poisoned input or a model provider having a bad day.

As a result, our final analysis has three layers. In the order that we built them (and that they often execute):

First model: panoptic mask2former

This is the same sort of computer vision model that is designed to power a self-driving car: it labels different parts of the street while also separating different objects from each other (so, for example, you get 2 different pole objects for different poles).

We started visualizing this output in a view like this: the yellow dot is the computed center of each object, and then we have a bunch of metadata about what the object is. For the sake of moving quickly, we chose to simplify each object into one point rather than try to cope with the full polygon data that the model computes.

The mask2former piece was the first piece we wrote. We let the team upload their own images and visualize the output in this UI, which was both very motivating for the team and the basis for great conversations.

Next: Gemini LLM

Working with urban planners, we wrote 100+ individual specific questions for an LLM to answer about the picture.

You can see some of the raw output of the LLM at the bottom. That first question (is_outdoor_streetscape) is the result of the prompt "determine if the main subject of the photo is an outdoor streetscape in an urban or suburban setting". In the table you will also see some summaries of different objects. For example, the section we see about the road ("This is a car-oriented...") is the result of us instructing the LLM "In less than 30 words, describe the street type and how it influences pedestrian movement. If car-oriented and applicable, mention a missing crosswalk".

We added some of these rules to our initial output-visualizer, which continued to create great conversations:

Three: Apply rules to stick it together

We apply a set of rules and heuristics to stitch the model outputs into an interactive visualization:

(You can see the interactive version of this example here)

While the Urban team had clear ideas for a more complex version, what we shipped amounted to show the Gemini summary for objects that mask2former recognizes, sorted with the most prominent objects first. Then when the user clicks "dig deeper" use a complex set of rules to figure out which piece of human-written urban educational copy to show.

For this street, StreetSmart calls out a lot of design features that make this street unpleasant to walk around. This is consistent with my memory of this intersection: it was not a pleasant place to walk or be, although it was a fine place to bike or bus through. I guess that's consistent with how the City of Cleveland describes these five miles of this renovated street: a complete street focused on bicycles, bus rapid transit, and car users.

Evaluating this structure

So even though our Gemini prompt contains phrases about the prominent bus stop in the middle of the road like "Despite bus stop shelters, the street lacks visible seating, water fountains, or public toilets for pause" for absence_of_any_street_furniture_utilities, we don't show that because we hadn't reliably found a way to tie that insight to a mask2former element in the picture.

Our current structure forces us to anticipate what will be in the picture and comment on it. When something's not in the picture (like the absence of street furniture), we don't comment on it.

But it also prevents us from making incorrect observations: For example, consider the bike lane on this street. In reality, Euclid Avenue has a bike lane painted on the road, but those lines are pretty faint. Our models tell us different things:

mask2former identifies a small bike lane (prominence 27), but it has low confidence in it (39% confidence)
Gemini is confident there is no bike lane, with the bike_lane_summary of "Absence of dedicated bike lanes on this wide, multi-lane street discourages cycling"

If we were simply looking at the Gemini output, we might confidently comment on the lack of the bike lane (which is untrue). However, because mask2former sees the bike lane as both very small and of a low confidence, we decide not to talk about it. There is assuredly more nuance we can derive from this kind of mismatch.

In conclusion

If I were starting again, I'd probably make similar core choices: boring tech, $0 infrastructure, strong guardrails on the LLM. What I'd do differently is be clearer to the team that a doacracy means real agency but also real ownership, and make sure everyone was on board with that tradeoff. I'd also reconsider whether tech that was boring to me was also approachable for the average volunteer developer, although I don't know that I should actually have written different code.

A mantra I hear about startups is that what gets you to one step probably won't get you to the next. If StreetSmart's next iteration does something functionally similar, the team will probably want to revisit several architectural decisions. Collecting reliable photo locations probably means shipping native iOS and Android apps. Exposing so much of the app's logic as client-side JS is fine for an MVP but irresponsible for a real tool. And the current backend has no systematic backups or rollback plan to the extent that a few strategic clicks could delete everything: that's a problem.

We made some choices. They got the project to the next step. The coming team will have to decide what gets them to theirs.

Nine Months of Unplanned PG&E Electricity Outages

2026-03-17T00:00:00Z

More than a year ago, I started to wonder about the reliability of my electricity in the Berkeley Hills: we get power outages a few afternoons per year, but is it really better a few blocks away?

I spent more than a year collecting house-level power outage data from a publicly licensed State of California dataset, and I've now put 9 months of it on a web map. While other data sources content themselves with showing information at the county level or regional level, we show more detailed shape-level information that is often accurate to the power customer level. Questions, comments, or collaboration requests? Send me an email at {my first name}@srcramer.com.

This document describes how I collected the data, sources of error, validation, information on the basemap, and a copyright notice.

Data collection and transformation

Step 0: CalOES gathers information

I download data from the public use licensed "Power Outage Incidents" from CalOES GIS Data Manager.

I assume the data reaches CalOES like this:

each utility has internal power outage detection methods which feed its internal dashboards
the utility turns this data into something fit for consumption by the outside world
CalOES has a server which receives or requests this data and periodically (every 15 min, according to the documentation) re-publishes it to their website

Step 1: gather data

Every 2 minutes, a server downloads JSON files from the CalOES map. The map publishes three data layers, of which we download two:

layer	meaning	action taken
0	Point information about where power outages are, which utility is responsible, how many customers are impacted, and some other utility-specific metadata	Downloaded but yet not analysed
1	Polygon information about what areas power outages cover, which utility is responsible, how many customers are impacted, and some other utility-specific metadata. PG&E is the only utility to provide polygon information for this layer, though in the future more utilities may	Downloaded and analysed
2	County-level summary information about the number of customers with and without power, the polygon shape of each county	Not downloaded (it's too many bytes for too little meaning)

The data gathering process downloads the files, computes whether it's seen the payload before, and then saves new payloads with a timestamp. If it's seen a payload before, it simply records that payload's hash and observation time.

Step 2: Compress data

A normal day of outages produces at least 1GB of text JSON files. Much of it is repeated: because outages last hours, the same polygon appears across many files.

To reduce long-term storage challenges, I compress these text files using Zstandard at compression level 22, which brings the average day to under 5MB.

The data produced can be really spiky. During December 2025 when SF had massive power outages, the raw text files ran to 12GB and the compressed data to 300MB!

Step 3: Ingest data

I download each of the days to my laptop and ingest the data into my local PostGIS server.

Step 4: Select data

For this analysis, I chose data that:

is in the polygon layer (as of publication, PG&E is the only utility providing polygon information)
is an unplanned outage (as determined by the utility metadata)
is an outage which the utility reports impacts two or more customers
was observed between 2025-07-01 and 2026-03-05.

Step 5: Compute polygons with the same outage performance

Power outages are unique because failures in the infrastructure tend to create a consistent pattern of outages. For example, when a power line on my street touches a tree, it will shut down power for all of the houses on that line until it's fixed. I could show standardized square or hexagonal bins which would give a sense of those underlying structures, but I wanted to be able to see them directly.

To compute those polygons for California, I first divide the state into tiles of 0.1° latitude by 0.1° longitude. Then, for each tile, I:

clip the geometry at the tile boundary
compute all of the polygon boundaries as lines using the PostGIS functions ST_ExteriorRing(ST_Dump(ST_Intersection(geo)).geom)
find all of the intersections using ST_Union(geo)
rebuild as atomic non-overlapping polygons using ST_Dump(ST_Polygonize(geo))
delete small polygons (small polygons can be generated when, eg, the utility slightly changes the boundaries of which transformers affect which areas)

Step 6: Turn the outage observations into hours

At its core we know point-of-time outage information, but we want to know duration information. As an example:

time	was the power on?
10:02 AM	yes
10:18 AM	no
10:28 AM	no
10:42 AM	yes

We chose to interpret this as meaning: "the power was out from 10:18 AM until 10:42 AM". In reality we know this is wrong -- even assuming the utility reports the information accurately, it's possible that the power went out anytime from 10:02-10:18, and that it came back on from 10:28 to 10:42. This interpretation hopefully means that variability in the start and end times cancels out, giving us a view that's at least internally consistent and also hopefully right on average.

As a practical matter, we:

grab all of the observation times (e.g. 10:18 AM)
compute the "next" observation time using PostGIS lead() (so that 10:18 AM row has a next observation time of 10:28 AM)
compute the time between the two observations (20 min), capped at 1hr

We cap the time between the observations at 1hr because the data collecting apparatus is prone to small failures and we don't want missing data to unfairly label an area as having bad power.

We then apply this back to all of the polygons:

compute an arbitrary point that's within that polygon using ST_PointOnSurface(geo)
select the power outages observations that apply to that point using ST_Within (e.g. 10:18 AM, 10:28 AM)
sum up all of the capped times between the observations (e.g. 16 min + 20 min)
apply that value computed at a point to the polygon which we know has the same outage information

Step 7: Simplify the shapes and put it on a map

To reduce the amount of data, I aggressively simplify the polygons. I found this sequence of PostGIS functions to do a nice job reducing lines which did not impact the polygon's shape (st_buffer( .., 'join=mitre')), separating polygons that are not actually connected (st_dump()), and communicating house-level detail of information (ST_SnapToGrid()).

(st_dump(
  ST_SimplifyPreserveTopology(
    ST_SnapToGrid(
      ST_Buffer(
        ST_buffer(geo, .00001, 'join=mitre'), 
      -.00001, 'join=mitre')
      , 0.00001)
    , 0.00001))).geom

Then, for every tile, Python generates a GeoJSON file that contains each polygon and how long its power was out as computed above. To make the map performant and hostable on Cloudflare Pages, Tippecanoe stitches those GeoJSON files together to generates PMT vector map tiles for various zoom levels. I then display them with OpenLayers, which manages to render it smoothly during panning/zooming in Chrome and Safari.

Sources of error

CalOES data does not match what's happening on the ground

For example, we see that parts of the Yosemite Wilderness had 193 hours of power outages. This is absurd: there is no electric service in the Yosemite Wilderness. We see a similar phenomenon around Lake Merritt, Tilden Park and assuredly more.

We also see inaccurate polygons in urban areas, like a polygon for a planned outage that covers 6 houses but where the structured information says only 1 customer is impacted. We excluded outages with only 1 impacted customer before we saw this example, but it stands to reason that there are more like it hiding in the data we show on our map.

Short outages are badly represented

CalOES publishes a point-in-time snapshot every 15(-ish) minutes. So if a property had a 5 minute outage, it's unlikely to be included. And if it is, it's more likely to be recorded as 15 minutes of outage.

CalOES does not update

On average I downloaded a unique version of data from CalOES every 12.3 minutes.

The longest intervals between datasets are:

Jan 23, 2026 from 12:52 AM Pacific for the next 8 hours
December 15, 2025 from 2:57 AM Pacific for the next 4 hours
July 13, 2025 from 6:04 AM Pacific for the next 2 hours

From what I can tell, these are caused by errors in the pipeline on CalOES's end where they are simply not updating their data for a while.

Downloaded invalid JSON

I occasionally download JSON that was truncated and as a result invalid. For example:

time (UTC)	hash	size	last few chars of file	valid
04:16:02	3352a	37 kb	`[-122.276971210187,`	❌
04:18:03	0da49	102 kb	`37.3675048846639],`	❌
04:20:01	02dc8	195 kb	`"IncidentId":"2748472"}}]}`	✅
04:22:02	02dc8	195 kb	`"IncidentId":"2748472"}}]}`	✅

For now I simply ignore the invalid JSON observations. As a result, I wouldn't record the observation until 4:20 even though it could be stale data from 4:16 or earlier. To me, this demonstrates the value of over-scraping this point-in-time data: I get a high-by-default temporal resolution, and when something goes wrong it can recover a bit faster.

My data collection infrastructure fails

During the time period I published in this map, I had no infrastructure failures. Because I'm recording point-in-time observations, there is little way to backwards-compute data that was not observed.

Outside of this period, I had a few scraping outages, most often servers running out of storage.

Outages caused by privately owned infrastructure

We have no visibility into events like a house blowing a fuse. Or an apartment building having a problem with its internal power distribution. Similarly, we have no visibility into recovery options like someone running a generator or using their in-home solar battery backup.

Validation

Having data pulled from the internet is good, but having it match up to ground truth is even better. Long story short - the data matched the other sources I found, and if you have PG&E I'd be really excited if you could take 5 minutes and help me validate more, particularly if you live outside the core Oakland/Berkeley/SF area.

Validating one outage at my house

My house had an outage on December 29, 2025. I found two sources of outage history for my house. PG&E power outage alert text messages and the export of hourly electricity usage, and they both tell the same story as the CalOES data.

PG&E text messages

I've signed up to receive texts about power outages at my house on PG&E's website. I have no memory of the alert texts being inaccurate, so probably they match reality? Indeed, they sent me three messages on that day:

1:33 AM - PG&E: Investigating potential outage near {your address},. More info to follow soon. Manage alerts: pge.com/myalerts
1:48 AM - PG&E: Outage near {your address}. We plan to have power on by Dec 29 08:00AM.
3:32 AM - PG&E: Power has been restored near {your address}. This was an unplanned outage. Still no power? Go to {our website}

These messages imply that the power was out for about 2 hours while I was asleep.

PG&E hourly power usage

On PG&E's website you can download your electricity usage broken down by either hour or (for some customers) 15 minute period. Interpreting the hourly electricity usage is a bit finicky. They don't always say "your power was out so your bill is zero". Instead, when the power was out, it seems to leave a note saying "This data was estimated" and is often accompanied by lower usage during the period. I saw:

Start Time	End Time	kWh	Note
00:00	00:59	0.35
01:00	01:59	0.16	* This data was estimated
02:00	02:59	0.15	* This data was estimated
03:00	03:59	0.15	* This data was estimated
04:00	04:59	0.29

Worth noting: There must be other reasons that the data can be estimated. One friend had their data estimated for a month, but I did not hear about any month-long power outage. So this needs to be taken with a grain of salt. Regardless, multiple records of consistently estimated data tend to match up with a power outage, so it's better than nothing.

In an ideal world a person would write a statistical model to notice that a bunch of power use was missing and use that as confirmatory evidence. But I've found my house's hourly consumption to be highly variable, so I'm not optimistic that it would tell me more than this quick eyeballing.

Data downloaded from CalOES

And in the underlying power outage data I see:

1:40 AM - no outage
1:50 AM - outage
[12 omitted observations] - outage
3:32 AM - outage
3:47 AM - no outage

And on the map

On the map we report 1hr 57 min of outage for that day. Which matches what we get for the PG&E text messages, and is also consistent with what we see in our hourly outage info (meaning that there was no power in the 1AM, 2AM, and 3AM hours). Always good to know that our data transformations are working.

Validating months of outages at my house

So I can make this little table:

Date	SMS	Hourly Data	CalOES
2/11/2025	outage	outage	outage
4/28/25	outage	outage	outage
10/19/25	outage	outage	outage
11/11/25	no	no?	no
12/29/25	outage	outage	outage

The only interesting example is 11/11/25. There, I had two non-continuous hours where the data was estimated. Maybe there were two small power blips? (I wouldn't expect to be able to identify those if they did happen). However, I'm inclined to believe that there was no longer outage if for no other reason than the 5AM hour in the middle has lower usage than either of the estimated 4AM and 6AM hours.

Start Time	End Time	kWh	Note
03:00	03:59	1.10
04:00	04:59	0.71	* This data was estimated
05:00	05:59	0.32
06:00	06:59	1.10	* This data was estimated
07:00	07:59	1.88

Validating with exported power usage

I asked four friends in Berkeley, Oakland, and San Francisco to share their hourly power usage from their PG&E portal. Their data exports mostly matched up with the CalOES data and what appears on the map!

My friend in Berkeley lives a few blocks away. Their confirmed that my house indeed has far more power outages than theirs.

Showing the outline of PG&E's service area

It was surprisingly challenging to identify where PG&E does and does not distribute electricity to end users. I could easily find this PDF map but struggled to find a representation of it that was suitable to import into PostGIS.

Instead I found this ArcGIS map which shows the outlines of a few utilities (including PG&E) from 2021-ish, mashed that together with an outline of California, and went about assigning approximate labels to the areas that were in the state but not covered by the major utilities listed.

In hindsight, I think I could have used this map of overlapping electricity distribution networks. I dismissed this map at first glance because it seemed obviously wrong (e.g. PG&E provides electricity in SF, not the City and County of San Francisco). However, I missed that there are overlapping polygons which can then be interpreted usefully.

I gained a greater appreciation for why it's hard to say which utilities cover which geographic areas: it's because large customers make special arrangements with different utilities, sometimes called releases. For example, most of SF municipal Hetch Hetchy Power's electricity is distributed via the PG&E grid (so the outage would appear in the PG&E data). However, it provides power directly to large projects like SFO airport, Salesforce Transit Center, 60% of SF streetlights, and a few SF neighborhoods (SFPUC pg 13). I generally stuck with the boundaries listed in the official maps, labeling SFO and Treasure Island as part of Hetch Hetchy Power, but leaving streetlights and Salesforce Transit Center as visually part of the PG&E area.

PG&E outages outside PG&E territory

The difficulty mapping PG&E's territory means that there are some places where PG&E shows an outage but my map data says another utility services the area.

For example, according to all the polygons I've been able to find online, PG&E does not service the Merced city center and the Merced Irrigation District does. However, there are clearly power outages reported near there, and the underlying data (which contains a "company" field) confirms that the company is PG&E. The town's website says that both utilities service the area, and Redditors suggest they have a choice between utilities at a given service address.

The interconnected nature of the grid means that PG&E might assign outage polygons to areas where they handle some middle part of the distribution. For example, the City of Alameda (which has its own municipal power distribution company) had an outage caused by a fire at a PG&E substation which supplies power to the municipal utility (source).

I opted to let the PG&E boundaries generally reflect the official maps, but still show the power outages under them. If you have any ground-level information about where people in these conflicting areas get their power, please do reach out so I can correct my visualization.

Optimizing the basemap

I opted to use vector basemaps because they make it easy to put the basemap on the bottom, the outage data in the middle, and the labels on top (thanks to the friend who suggested it).

I used a few mostly-monocrome basemaps, but chose a colored one so that the user can clearly see green nature areas and blue water (which rarely have power coverage and as a result don't have power outages). I started with the Bright style from OpenFreeMap.org and then edited the stylesheet to remove any of the yellow-ish colors which could be confused for a power outage (e.g. yellow roads, sand) and a bunch of lines (e.g. county-level boundaries, ferry routes, etc.) which are unlikely to help the user orient to power outages.

Future work can include re-computing these map tiles so that they only include the elements I render, further helping lag when panning/zooming on the map.

Copyright information

Basemap data is © OpenStreetMap contributors.

Service area polygons as presented are © by the author.

Sunset Here is Sunrise Where?

2025-12-17T00:00:00Z

When it's sunset in San Francisco, it's sunrise in Bhutan. In June, it's sunrise in Amsterdam instead. The pairing changes every day, tracing an arc across the globe as the seasons shift.

This felt like something Google should be able to show me, but couldn't. Neither could ChatGPT. So I built a map that can:

Eyeballing an answer

As I Googled, I realized that I could kind of compute where the sun is simultaneously rising and setting with a map that shows where sunshine is.

The map on CelestialProgramming.com plots a line showing the edge of sunlight, which the world calls a terminator line. On December 4, I could move the time slider to approximately where the sun is setting in California. Then I could look at the arc of where the dark transitioned to lightness and see that it was rising ... somewhere in Bangladesh? And I guess also somewhere in China and Eastern Russia?

A quick Google suggests that San Francisco's sun sets at 4:51 PM in GMT-8 and sunrise in Dhaka, Bangladesh is 6:26 AM in GMT+6. That's a difference of 25 min: close enough to make a good story, but far enough to try to do it more accurately...

Playing around a bit more I noticed that the corresponding sunrise fell on the other side of India! So the answer would change throughout the year...

Enthralled, I set about to make the insight of this small map more accessible. This code is public domain licensed and Claude trivially bolted on an interactive map so users could pan and zoom around the world. But I then wanted to make a story out of it, so I set about trying to turn it into memorable sentence. This is where it got tricky.

Getting places around the world

To start building this memorable sentence, I grabbed a few thousand large cities from Wikidata, Wikipedia's structured-data sibling.

But as I started playing rigorously, I noticed that there were times when the sunrise was happening in the middle of the ocean. What can you call out then?

I turned to Wikimap, which plots Wikidata on a map. I saw the ocean was filled with:

shipwrecks. The ocean has a lot of mapped ship wrecks. But I'm trying to show something happy, and no one wants to think of dead sailors at sea. Rejected ❌
solar eclipses. Wikipedia has hundreds of years of eclipses, mapped to the coordinates of the maximum eclipse. This struck me as boring. Rejected ❌
airports. WWII resulted in a lot of airports on tiny islands, and my target audience (my dad) loves planes. These would be a keeper ✅
seamounts and undersea canyons. I'd never heard of these, but they sounded cool. They tend to be in the middle of the ocean with nothing else nearby. Keep ✅

I also ruled out:

small towns. While it would be nice to include small towns with no airports like Edinburgh of the Seven Seas, there are 3 million civilized places and Claude couldn't write me a Wikidata query to easily extract them. I passed ❌
islands. While they would include atolls like Nukuoro, the world has a lot of islands. The islands one might consider important probably already have an airport, and an island like Greenland is too big to be useful here ❌
reefs. Reefs are larger than a single point. It's not legible to say "the sun is setting over the Great Barrier Reef" -- it covers 1,400 miles ❌

Sticking with just large-ish towns, airports, and seamounts, I came up with a list of 50,000 points that cover most of the planet. And indeed, on that day when it's sunset in Sydney, it's sunrise at Corvo Airport in the Azores and Tanquary Fiord Airport in Nunavut, Canada.

Finding places that mean something to someone

50,000 points of global coverge is great, but is too many for our visualization.

First, I had to decide the density of points I needed. On the day I was checking, San Francisco's sunset was about 5 minutes different from Stockton's (60 miles away); close enough to not need a separate point. But SF to Yosemite (150 miles) differed by 10 minutes, which felt like it should have a separate point. I settled on selecting points at least 100 miles apart.

Then, I'd need to rank the points. Ranking cities by largest population seemed obvious, but that seemed fraught: in the San Francisco Bay Area that would mean we call out San Jose and not San Francisco! Claude suggested that I show the city with Wikipedia articles in the most languages, and this worked well: San Francisco is covered in 80 more Wikipedias than San Jose.

I decided on a final ranking algorithm. The points that were the highest rank within 100 miles would go to the front end, and the UI would call out the highest ranked point for a given set of sunrise/sunset cities:

cities first, ranked by number of Wikipedias, then population
airports next, ranked by number of Wikipedias then annual passengers
seamounts last, ranked by number of Wikipedias then depth

I discovered an emergent behavior I hadn't anticipated: the 100-mile rule chains. South Lake Tahoe is within 100 miles of Sacramento, so Tahoe gets excluded. Sacramento is within 100 miles of San Francisco, so Sacramento gets excluded too. This chaining found a good distribution of places throughout the world without needing hard cutoffs.

It works well. Thinking back to that troublesome sunset in Sydney, I've never heard of either Corvo or Tanquary Fiord, but Corvo Island has a Wikipedia article in 30 languages, whereas the Tanquary Fiord article is only in 6 languages. Probably calling out Corvo will mean more to more people.

Building with Claude

I used Claude Code throughout, and it handled the drudgery beautifully. Tasks like "move the program logic from old.html into the UI in new.html" were as simple as typing that sentence. It wrote Wikidata SPARQL queries with their arcane identifiers (wd:Q165357 instead of submarine_volcano). It imported CSVs into Postgres and exported query results to JSON, even handling details like rounding coordinates to 5 decimal places. But there were still some moments where I was glad I'd spent a few years doing it the hard way.

At one point, Gemini had grouped the UI elements into a sidebar div that worked on desktop but broke on mobile—the sliders appeared directly below the sentence, mid-screen, instead of anchored at the bottom where a thumb could reach them without covering the map. This was a UI problem because it meant the user's thumb would cover the map as they moved the sliders. Usually finding the bug is the hard part, but I spent several turns insisting Claude fix the responsive layout; it kept failing. After a bit of old fashioned debugging I suggested pulling the elements out of their parent div, it worked immediately.

Similarly, Claude needed help with Wikidata SPARQL extraction queries. Claude wrote the structure and successfully found common items like cities, but wasn't a help when the list came back with permafrost and not undersea mountains. While I'd never worked with Wikidata queries, I spent a few minutes debugging and realized Claude had found the wrong Q-number. I pointed this out with the correct example, Claude gave some Claude-isms ("Q179918 is permafrost, not submarine volcano!"), and proceeded to get the answer I expected.

Claude helped me hack the initial version in a day, then I spent a few more days playing with it and iterating on feedback from me and the friends who tried it. I added some more labels, will make the touch targets a bigger, etc. It's an exciting time to build.

Conclusion

I keep coming back to the map, sliding through the year. I had no idea Bhutan and San Francisco shared a sunset line, or that it would drift to Amsterdam by summer. The world is more connected than I thought.

You can find the map here.

Finding the City Based On What's Open Now

2025-04-08T00:00:00Z

What's a city? There are lots of definitions. A Country of Cities thinks it's a place with 30+ dwellings per acre. Some new urbanists think the answer is a "15 minute city" where a person has most of what they need (food, school, park, etc.) within a 15 minute walk or short transit ride. My rural dwelling relatives might think about a traffic jam and muggings. When I was a kid I thought it was all about tall buildings. But now when I think about a city, the thing that makes it unique is people. Visible people. It's a place with a lot of visible people.

But how do you find these people? I found the existing solutions frustrating, so I decided to use my newfound GIS skills to play with a new proxy: where is open now? I came up with a map of places that I can consistently go and find visible people (demo here, screenshot below):

Similar Work: Areas of Interest in Google Maps

In 2016, Google Maps launched a new feature to show areas of interest. You've seen these areas before - it's the orange-ish area that is the background of places that have a lot of stores and places to eat. This feature is a good place to start exploring a new city, but still doesn't help me understand where I want to walk around right now.

The feature has a few main flaws:

It shows the same shading at all times of the day. This might be accurate for 24/7 places like Times Square, but is not accurate for most places where Americans live. Time of day clearly matters.
It thinks of parking as interesting space. For example, a nearby mall's expansive parking lots show up as "areas of interest". In reality this mall has a few interesting places to walk surrounded by massive big-box stores with few entrances and dead parking lots.
It does not consistently identify less advantaged areas as "interesting" (at least according to Bloomberg when the feature launched). We should have a way of finding people regardless of whether we think we'll like them.

Perhaps this is a fine way for people to think about where they might want to drive and walk around, but it's not a useful way for me to figure out where I should walk around now.

Using Open Establishments to Figure Out Where People Are

But what if we could measure street-level energy by simply looking at what’s open?

I think that establishments are a good indicator of human-scale, street-level activity. These restaurants, cafes, bars, libraries, museums, art galleries, etc. interact with the urban form around them, shaping it as they are shaped by it.

I hypothesize that:

when there are a lot of people in an area, establishments will (try to) open to serve them
when an area no longer has people, establishments will close because there are no customers
when there are multiple open establishments with their entrances really close they will generate street traffic as people walk between them
concentrations of establishments draw more people to them

This will break down for some areas:

parks and large public spaces (like a town square) can draw lots of people, but can be so large in time and space that it's hard to understand which parts are interesting at a given time
schools, religious establishments, stadiums or conference centers can generate a lot of traffic at certain times (e.g. when school lets out), but are unlikely to be interesting to a passerby not involved in that institution

But even with these limitations, perhaps looking at open hours will be more useful?

Comparing Open Areas to Google Maps

I've found that making something visible can help us explain what we love (or hate) about it, so I started with my home area of Berkeley, CA to see how this could work.

We can compare this "open now" visualization to Google Maps.

To me, the lower map better represents Berkeley:

there are four main commercial avenues with lots of businesses
each of these avenues has little dead spots in the middle - there is no continuous commercial area that's pleasant to walk along
there are little pockets of businesses spread throughout the middle meaning most Berkeley residents are a short walk from a pedestrian area
there are no concentrations of businesses in the hills (the top right corner where all the roads are not straight)

My visualization is even more accurate when we see Berkeley at 11 PM; almost all of the streets are quiet.

So how did I build it? Read on...

How I Built It (the fun part)

One: Get the Data

Just like cities, which establishment is where and open when is constantly changing. Big players like Google or Yelp spend a lot of money to have this accurate and current point of interest (POI) data. This data is also useful, so they charge for it.

The best free POI data with open hours is on OpenStreetMap (OSM). POI only tends to be accurate and complete in areas where there are dedicated volunteers mapping the area. Using OSM data means that I will only find open businesses in areas that people have ambitiously mapped. Probably this is okay - I'd expect more people are more interested in mapping areas that are fun to walk around.

The incompleteness of the data also risks under-representing communities. The main barrier to editing OSM data is (a) knowing that OSM exists, and (b) spending the time to read the documentation and learn how. Anyone with time and a smartphone (or library computer) can improve their neighborhood and its representation on a map like this.

Once that was loaded (details below), I could play with my algorithm.

Two: Find what's open

As a sample of our analysis, consider an example of a very local shopping area near Berkeley's Monterey Market.

In 2025 Google does not show this as a "Area of Interest" when zoomed out, but on a Saturday it's a hopping place to see lots of other people.

We take the raw map data provided by OSM and overlay with a red dot the businesses that are open at 12:15 PM on a Saturday (look closely; the red dots are subtle):

We see that there are a lot of small businesses open next to each other near the Hopkins Bakery, then there is the Espresso Roma across the street, and Monterey Market across the other street. We see that the Hopkins Laundrette and Berkeley Horticultural Nursery have no dots - they are in fact open, but we don't have data about their open hours 🤷‍♂️.

I imagine each open place generates a little gravity field—about 100 ft of liveliness around it:

We see that the small businesses along Hopkins form some overlapping circles that all connect into one blob. However, Monterey Market and Espresso Roma are just a bit too far way to really be in that blob.

This matches my experience on the ground: combined with 40' wide street, the parking of Monterey Market and Roma Espresso's front garden prevent them from feeling connected to the busy street life on that southwestern section of Hopkins.

Four: Cluster overlapping circles

We can then take these intersecting circles and join them together:

We now see three blobs of open businesses: one with those businesses on the Southwest corner of Hopkins, one for Monterey Market, and one for Espresso Roma.

Four: Isolate areas where 3+ circles overlap

If three or more businesses overlap, it’s we hypothesize it's a social spot. So now we simply look for clusters with 3 or more establishments:

Five: Scale across time

As a simple SQL query, we simply add group by openTime to our queries to compute these shapes at different times.

We see that these businesses all open around 10 AM Saturday and close around 6 PM Saturday. Indeed, this block is a fun place to be during the daytime, but is quiet at night.

How I Built It (the technical tricks)

Delivering novel results is easier when you pick the right tools. Here are some of the technical tricks I used.

Load OSM data in to PostGIS

I love SQL, and find it intuitive to manipulate GIS data in PostGIS. I first downloaded OSM data in the .osm.pbf format from geofabrik.de. Then I loaded it into PostGIS with osm2pgsql using a custom lua configuration to store almost all OSM data attributes in JSON.

The terminal command is something like

osm2pgsql -d osm --output=flex --style=/path/to/osm2pgsql-flex_config-generic.lua ~/Downloads/norcal-latest.osm.pbf

Interpreting OSM open hours

OSM open hours are stored in a somewhat-freeform format like Tu-Sa 10:00-16:00.

To structure that, I used the opening_hours evaluation tool to ask whether the business would be open at 2024-05-07 10:15, 2024-05-07 11:15, etc.

I chose to ask about times like 10:15 AM rather than 10:00 AM because many businesses open at these sorts of round times (like :00) and I didn't want to worry about the boundary condition (e.g. if it opens at 10:00 AM, will it show in the data as open at 10:00 AM?).

Since I only wanted to do this analysis once, I did a select distinct of all the open times and used the Chrome interactive terminal to get an open answer for all of the time strings at my sample timestamps. Then I loaded that into PostGIS and joined it onto my existing data.

Here was the simplified JS I put in the page

// https://openingh.openstreetmap.de/evaluation_tool/
var times = ['2024-05-07 10:15','2024-05-07 11:15' /*more times to model */]
var open_hrs = ['24/7','Mo-Fr 07:00-16:00; Sa, Su 08:00-15:00' /* more times from OSM */]
var outs = []
var doOneTime = function(h) {
	try {
		var oh = new opening_hours(h); 
		times.forEach(function(t) {
			outs.push({
				opening_hours: h, 
				obsTime: t, 
				isOpen: oh.getState((new Date(t)))
			})
		})
		outs.push({
			opening_hours: h, 
			obsTime: 'weekStable', 
			isOpen: oh.isWeekStable()
		})
	}
	catch(e) {console.log(h);}
}
open_hrs.forEach(doOneTime)
console.log(outs.map(i=> i.opening_hours + '	' + i.obsTime + '	' + i.isOpen).join("\n"))

Working with data in PostGIS

I once wrote thousand-line programs using SQL Server Management Studio and grew accustomed to having an interactive SQL environment.

To recreate a similar environment with GIS visualization, I use DBeaver. I simply write select st_union(geom) from my_table; then click the spatial button to see my results.

To visualize the results, I use Leaflet with Leaflet Timeline starting with the Leaflet Timeline Earthquake demo.

Paid services exist to serve a beautiful background map, but for now I make due with the default map from OpenStreetMap. I make the background map 80% greyscale to help the open areas pop out while still distinguishing important colors like roads vs rivers.

Reclaiming My Media Diet with RRSS - A Custom RSS Reader

2025-03-22T00:00:00Z

As I transition to a career in climate tech business operations I wanted to read more about the climate tech landscape. Like many 30 year olds, I entertain myself by scrolling on my phone. Lots of people scroll social media, but I scroll something far more addictive: Hacker News (HN). The site is a list of tech news articles that entertains me while also letting me pretend I'm doing professional development.

I love the format of Hacker News, so I wrote a private news aggregator that has a similar interface for content that I curated.

I eventually came up with this, my custom RSS reader:

screenshot of RRSS, the tool I built

Why I love Hacker News

This new tool will replace HN in my flow, so I started by thinking about why I spend so much time there. For reference, here is how it looks in 2025:

For starters, I like the content on the site. It is:

relevant for my professional skills - In addition to being fun, it's helped me think and talk about the tech landscape more broadly.
apolitical(ish) - The political sphere is always important, but keeping up with the latest political outrage is exhausting. The articles on HN tend to focus on the impact to business or people (e.g. "Google Maps now shows the ‘Gulf of America’"), not the outrage ("Almost Nobody Wants to Call It the ‘Gulf of America’")

I also like the HN user interface. It is:

fast - Typing a quick ne is enough for my browser to know where I want to go. The site loads quickly--at less than 1 second it's easily the fastest site I use.
information dense - The visual design emphasizes the article's title. The lack of other visual features like pictures put a lot of articles on the screen and make it very easy to skim. Supplementary details are carefully chosen and provide additional context. For example, it's helpful to the article's website because my expectations are different for an article on ieee.org, twitter.com/benswerd, and deepmind.google).

However, it has a few downsides:

never ending - I've decided that reading to the 300th entry is enough, but there's no satisfying way to be "finished" reading HN.
inflexible - I want to read more of the premium publications I pay for as well as climate tech industry news. There's no good way for me to add that content without influencing the HN hive-mind which submits and votes for which articles receive visual prominence.

What to read instead

I had an exciting opportunity to think about what I want to read.

I wanted content that was:

interdisciplinary - My interests include tech, startups, established businesses, climate, energy, industry, information security, and the world around me. I want to read about all of them.
global - The world is a big place, the American media landscape has problems, and I can get around some of that by reading international publications.

I talked with a lot of people in climate tech about how they keep up with the space. People recommended environmental coverage in regular news outlets like FT, The Economist, and Bloomberg. They also suggest a few specialized climate news outlets like Canary Media, and Latitude Media.

Designing a replacement

I considered a few different solutions, namely newsletters and news apps themselves (see rant in appendix), but found they didn't hack my brain in the way HN does. I set out to build a replacement.

User interface goals

The user interface goals mostly amount to "copy the good parts of HN". I want the UI to be:

fast - It should load quickly enough to be habit forming
skimmable - I should be able to scroll it faster than a Facebook feed
finishable - On a slow news day, I should be able to feel like I've read all there is to read and can put the phone down.

Getting the news via RSS

HN solves the problem of "what to read" by asking its users to submit articles. My tool will only have one user, so I needed another source.

A classic solution to aggregating different news sources is RSS. In addition to their front pages, most news organizations (and many blogs) also provide a data feed of the news in a machine-readable format. Here is an example of the RSS from the local SF Gate. Anyone can collect these data feeds from around the internet and present them in any way they'd like -- it's their benefit to users and threat to web conglomerates.

RSS's heyday was in the 2000s and its decline was cemented with the 2013 sunsetting of Google Reader, widely recognized as the class-leading way to consume RSS feeds. As a result, not all publishers always provide RSS feeds, but many of the reputable ones still do, and there are some workarounds to get feeds for the publishers who no longer appear to provide them.

Designing and Building the RRSS Application

With my goals in mind, I could design the application. I wanted to iterate quickly using a tech stack that was easy to deploy. Here were the parts:

index.html would be my feed reader with a layout that accomplishes my UI goals. JavaScript in that file would grab a JSON blob, do any final adjustments, and then present it to the user
load.php would loop through the RSS feeds I wanted to aggregate, tease apart the XML, and write new entries into my SQLite database. I chose PHP because the code shouldn't be very hard, my Mac already runs a PHP development server, and I have terrible memories from deploying Python applications.
rrss.db would be an SQLite database that keeps state. I prefer SQLite over other file schemes like JSON because I find SQL data manipulation easy and universal. I decided to keep all the data in one table that has a few columns: url (the primary key, to prevent duplicates), publish_timestamp (what we sort on), and txt (a JSON blob that represents the article as it appeared in the RSS feed).
rrss.php would grab relevant articles from rrss.db and serve them as JSON to index.html
rrss.json a later creation, but simply the output of rrss.php written to a static file

All things considered, the application was pretty easy to develop. I used the standard PHP XML library and wrote some custom logic to ensure most of my RSS feeds parsed successfully.

Deploying RRSS on a Free PHP Host

I chose PHP because it's often super easy to deploy (drag files to server, be done). However, finding a free spot on the internet turned out to be the most challenging piece of this project. I wanted to visit example.com/rrss from any browser and see my feed!

I can easily host on GitHub Pages or Cloudflare Pages for free, but that only works for static sites. I wanted to refresh my RSS feeds periodically (I eventually decided on every 5 minutes), and that usually requires a more expensive host.

I hosted it for a few months with InfinityFree in 2024, but stopped when their service stopped including Cron (a way to automatically run my update code every 5 minutes). I tried again in 2025 thinking that I could have one of my other computers run cron and simply curl load.php, but I discovered that InfinityFree actively stops requests from Curl.

I stumbled on Oracle Cloud and set up their free hosting. It was more challenging than I'd have expected - I wrote a small guide to help others. Once I figured out Oracle's networking, I then had problems running my application code:

I needed to install PHP extensions for features I thought were built in like XML and SQLite.
PHP had trouble reading/writing to the sqlite file when it was called via Apache. To get around that, I wrote another script that runs via a PHP command line and writes the output of rrss.php to rrss.json so that it can avoid the Apache -> PHP -> SQLite sequence.
Then I tried to handle all the PHP execution via Cron instead and found that I needed to more completely reference the path to my database file (PDO("sqlite:".__DIR__."/rrss.db"); was successful; PDO("sqlite:rrss.db"); was not).
I changed from doing the HTTP request with file_get_contents() to curl_exec() because file_get_contents() on Oracle could not successfully get the feed from Financial Times even though it worked on my Mac.

Compared to the simple experience I've had running simple PHP scripts on my development environment and shared hosts like InfinityFree, I was surprised at how much I had to configure the Oracle VPS.

But that's finished now, the code is a bit more robust, and for now I have a working product! Check it out (for now) at cloud.srcramer.com/rrss.

Run it yourself!

You can download the code on GitHub

Appendix: Finding RSS feeds

Another roadblock was finding RSS feeds. Back in the heyday of RSS a browser button surfaced RSS feeds, but that no longer happens. Here are some places to look:

the website footer - look for it next to the social media icons or in a link called "feed"
Search - Googling "{org} RSS" is likely to turn up an ancient page listing where the feeds used to be, like the BBC's documentation from 2011
Reddit - Some sites like NPR have feeds, but the documentation is a Reddit post
add /feed - platforms like Wordpress or Substack have feeds, but you need to guess the pattern. Like construction-physics.com -> construction-physics.com/feed.
Google News - formats search results as an RSS feed when the URL parameters are like https://news.google.com/rss/search?q="cleantech"

Appendix: Why I dislike the alternatives

Building tools is a fun way to get exactly what I want while learning something new, but it often takes longer than I'd expect. Here are some alternative ways I considered to meet my same need.

Newsletters are hard to skim

I think there are two main categories of newsletters. Those that contain:

one piece of original content, like the one from Bits About Money or Doomburg. It's not reheated, it's not published anywhere else, and I can tell from the subject line whether I want to read it.
"curated" links from around the internet

I love the ones with one piece of original content. The subject line works like an article title (because, often, it is). The first words tell me what it's about. The entire thing is designed for skimming.

I have trouble with the other type. I find it too hard to skim. For example, which would you rather read: this synopsis of a news article buried in an otherwise great newsletter...

🌊 London-based geospatial analytics startup Ocean Ledger raised €900,000 pre-seed funding to scale its AI-powered coastal risk forecasting solutions, which use satellite imagery and machine learning to help engineering firms, municipalities, and insurers manage climate-driven shoreline threats.

...or the title of the news article itself from its publisher...

Ocean Ledger secures €900k to increase accuracy of coastal risk management

I'd rather read the title of the news article. Or, ideally, both: one piece of information with the succinct title and Climate Hack's keyword-stuffed description. Typically, just reading the headline will give me enough info to decide if I want to read the article.

Many of the newsletters with one piece of original content already have an RSS feed to integrate into my RSS tool. Since a lot of good climate links are shared via "curated" link newsletters, I’d like to build a scraping tool that will extract links from "curated" link newsletters for data sources that do not have an RSS feed.

Newsletters arrive on the wrong cadence

A newsletter makes sense when it's something that I want to commit myself to reading. But it doesn't do well for that awkward gap of "I'm bored this second, help me find something informative and useful right now". Instead they're mixed in with all the other work I want to not think about in my inbox.

To make matters worse, the curated links newsletters suffer from their delay in publishing: the author needs to read the article, wait for the next newsletter, and then send it in the next newsletter.

Too many publisher apps

I could also simply visit all of the sites that contain content I find interesting. But there's simply too many to remember.

One of the exciting parts of my professional skill set is that it's really broad. I want to keep up on the latest developments in programming, information technology, business, geopolitics, energy, and cleaner ways to make physical products. The best nuggets are buried in the pages of technical publications, not on the headlines.

Also, a polling strategy could over-poll top-of-mind news outlets (like NYTimes) but leave me forgetting about other useful perspectives (like Der Spiegel).

Read a physical newspaper

Oh that would be great. I need something a bit more addictive, though, to compete with Hacker News.

Power Outages in SF Bay Area (Part 1)

2025-03-09T00:00:00Z

I never thought much about power reliability—six years in downtown Chicago, and I can’t recall an outage that impacted my life. But in the Berkeley Hills, nestled in the wealthy, high-tech San Francisco Bay region, we lose power a few afternoons a year. It’s baffling that homeowners in $2M houses tolerate sudden WiFi, fridge, and light failures. During our third power outage in as many months, I wondered: How reliable is my electricity? Is it better a few blocks away?

Averages Don't Tell It All

Power outages don’t follow city or county lines: when the power goes out at my house, we walk 10 minutes to a café and keep working.

I imagined a map showing power reliability across my area—something like this:
Author’s recording of power outages within 10km of Berkeley between November 22, 2024 and December 10, 2024. Dark orange is a lot of time without power; grey is a bit of time without power; no color is sweet 100% reliable electricity.

Surely someone must publish this map. So I went looking for it online...

Currently Available Outage Information

Real-Time Outage Information from PG&E

In my neighborhood, power outages tend to be highly localized. My intuition is backed up by actual data.

PG&E, our flawed utility provider, it shares the exact location of houses without power. For example, as I type this now, about 1 customer in this area of approximately 15 houses has a power outage due to an unplanned outage. Their neighbors have power.

The homes in the green colored area have their power out, and have for the last 5 hours. I’m sure someone writing a book on PG&E’s organizational culture can write a nice chapter on why this area is colored green.

Historical Outage Information from PG&E

While PG&E publishes this high-resolution outage data, historical performance is reported at a level approaching the county-level:
https://www.pge.com/assets/pge/docs/about/pge-systems/CPUC-2023-Annual-Electric-Reliability-Report.pdf

This is cute; it clearly doesn’t tell a good story, but it also doesn’t tell the full story.

They also have a data request program, which allows academic researchers and governments to request data. However, the log of past requests and text of data available suggests that polygon-level outage information is not available.

Outage information from Poweroutage.us

The private market has come to the rescue, offering outage information for the entire country.

https://poweroutage.us/area/state/california

Their product page boasts: “Need historical data? We've got it! Every data point collected is also archived! Outage data can be retrieved at the utility, state, county, and city levels. Data can also be summarized in any way that meets your needs, raw data, hourly, daily, outage events, etc”.

Inquiring about purchasing polygon-level data reveals that they can’t provide anything more granular than the city/county level.

Outage information on GitHub

User Simon Willison scrapes PG&E’s outage map and posts a history of it on GitHub!

It seems promising, but a look through the readme shows that Simon’s data doesn’t have any polygon information. Per his comments:

This repository only archives outages that are reported for a single location.

The outage map also includes polygon data, which is much more interesting... but has not proven practical to archive here, for reasons explained in this issue comment. Short version: I'd have to constantly archive 100MB of data per snapshot because the polygons are so large!

It also suggests that people aren’t capturing this information because it’s too much data. Hmmm…

Outage information from California Office of Emergency Services (CalEOS)

The State of California essentially re-publishes the outage maps from PG&E and several other large California utilities on ESRI’s ArcGIS Hub.

This is the juicy data I’m after! From a state agency! Surely they’re capturing historical data!

Alas, an email reveals they are not capturing historical data. However, there’s both a “download data” button and an explicitly labeled Public Use license.

Perhaps I can make a contribution here…

Putting the pieces together: scraping CalEOS’s data

Spread across three files, CalEOS shares 11mb of data (approximately 2 iPod songs of data) updated every 10-20 minutes.

Indeed, this is a lot of data. Storing the raw data in GeoJSON would take something like 1 gb per day – enough that 2 weeks would fill a Gmail inbox on the free tier.

As I designed my data collection and analysis workflow, I had a few goals:

Start quickly. Time marches forward, and every 10 minutes that passes is 10 minutes of unanalyzable history.
Run locally. I’m cheap, so running the workload on the Intel NUC computer in my closet is more attractive than paying $48/year for a VPS. If I want it, I can achieve greater reliability by putting an old computer in a friend’s closet.
Maintain high resolution. Low resolution data already exists, so high resolution data is the main differentiator of this project.

This lead to a few decisions:

Build a dumb scraper. I love SQL databases and eventually want all this data loaded into PostGIS, but I could write my scraping code now and then worry about my analysis and importing code later.
Be (slightly) reckless with storage. Since I could run it on my old computer with 100gb of storage, I could simplify my program by not over-optimizing the storage consumption. I can balance the work by only recording the body of files that I’ve never seen before, and then figuring out further deduplication later. This is another benefit of running locally - storage is a key price differentiator in the VPS world.
Use basic Unix tools. Since the scraper was pretty dumb, I don’t need to worry about setting up and maintaining Python, PostGIS, ZFS, or whatever else on my scraping machine. Curl, cron, shasum, and the file system can do the heavy lifting.
Use immutable data structures. Nothing should delete or edit data that’s been written before: it will record facts, and then the eventual analysis will turn that into a story.
Use hashes to manage duplicates. The data from CalEOS only changes every 10 minutes or so, but I want to be able to estimate timing to the highest resolution available. As a result, to more accurately estimate the change time, I need to download the data more frequently than 10 minutes. This could balloon the storage required by downloading tons of repeated data. To manage that, I’d compute a hash for each file, record the payload of the file when it’s new, and record the hash and time when we’ve seen it before.

Because I wanted to use basic Unix tools, I needed to think a lot about files. For simplicity, the filename does a lot of the heavy lifting, so it’s the first thing to design:

2024-12-02/layer0-2024_12_02T05_20_02_utc-0dff5…136f.json

2024-12-02 - to simplify human management (and–eventually–file system performance) we create a new folder for each day.
layer0 - CalEOS publishes their data across 3 ArcGIS layers, so record which layer this is.
2024_12_02T05_20_02_utc - the UTC timestamp of when we downloaded this file. We use a human-readable date because a growing application will need some human file management. We use UTC so that multiple computers in multiple time zones have a chance of collaborating successfully. We put “utc” in the file name because I don’t want to worry about time zones in the future.
0dff5…136f - the hash of the file. Hashes are a way of uniquely identifying a larger piece of information: to a reasonable approximation, when the file changes, the hash of the file changes.
.json - tells me that the file contains the GeoJSON payload of data, rather than simply being a file describing that we’ve seen this hash before. (use .info if it does not contain the GeoJSON payload)

With the key pieces written down, we can now write our bash function to download a file:

Change to the proper directory, creating it if it doesn’t exist
Use curl to download the file to a temp file
Record the time the download finished
Compute the hash of the downloaded file
Write a file to disk using the naming scheme above. If we’ve never seen the file, record the entire file. If it’s a duplicate, create an empty file to record that the data has not changed.

Then, we loop that across all of the URLs we want to download, tell Cron to run it every 2 minutes, and we’re off to the races!

Check back for updates about analysing this data!

Where can you fly? (2019 data)

2021-02-28T00:00:00Z

I knew that New York was well connected to Europe. I knew that LA was a gateway to Asia. I'd read it in airline industry publications, seen it referenced in news articles. But I'd never seen it.

The time was 2021, the middle of the COVID lockdowns. I hadn't traveled anywhere in a while. I love planes, I love thinking about how they connect the world, and I wanted something to stare at. I discovered that the US government publishes data about every commercial flight that touches a US airport, and I turned it into an airline-agnostic route map of where people could fly in 2019.

Each line means at least one large plane per day worth of seating capacity flies between the two airports. Thicker lines mean a stronger connection: bigger planes or more planes per day. The top 50 connections are in red, the rest are in blue. See the full map here.

The data

The US government publishes data about where commercial planes fly when those routes touch the US.

This data match takeoffs and landings when one of those occurs in the US. It does not show what a person intends to fly. Consider three ways of getting between Chicago and Paris, and how that appears in the data:

Chicago ORD -> New York JFK -> Paris CDG, the data records two flights: one for ORD->JFK, and one JFK->CDG.
Chicago ORD -> Frankfurt FRA -> Paris CDG, the data records one flight: ORD->FRA. The flight from FRA->CDG is not included because it doesn't touch the US, even if it was purchased on one ticket.
Chicago ORD -> Paris CDG, the data records one flight: ORD->CDG.

While airline marketing departments have taught us to think of these flights as the same because they all start in Chicago and end in Paris, this reality of routing matters: beyond the obvious differences like scheduled travel time, some of these routes are more resilient to bad thunderstorms or overrunning meetings than others. This data reflects the deep-pocketed possibilities of a savvy passenger in our air network, not the marketing-department illusions.

Also, the data reflect seats available rather than passengers flown. So if a 300 person plane flew between ORD and JFK once, it wouldn't matter in this data whether it was fully packed with 300 people or virtually empty with 3. We're mapping deep-pocketed traveler possibilities, not airline profitability.

Visualizing the data

While a Wikipedia airport page shows what routes are possible, we focus on which routes are common and plot it on a map.

Commonality matters. For example, it's fun that one can fly direct from NYC to Singapore, but only 58,443 seats made that trip in the entire year. That's enough for one (small) planeload per day. Conversely, 1,800,000 seats flew between New York JFK and London Heathrow in that same year, enough for one (big) planeload per hour between 6AM and 9PM every day of the year. We want to see that there's a sky train between NYC and London, and we'll leave that direct flight to Singapore for the trivia books.

Maps matter. We see that some regions are better connected than others, and maps allow us to quickly see that.

First takes from the top 50 air routes

On the map, the top 50 routes are shown in red.

New York - London outranks almost every domestic route. More seats fly between London LHR and New York JFK than all but three US airport pairs (LAX-SFO, JFK-LAX, LAS-LAX).

Western Europe and East Asia are well connected. Large international hubs like Paris, Frankfurt, Amsterdam, Tokyo, Hong Kong, and Seoul all have thick lines to multiple American cities.

Our connections south are thin. This is partially an artifact of how we project maps, but our routes to South America, Central America, Africa, and Australia are strikingly sparse compared to the dense web going east and west.

NY goes to Europe; LA goes to Asia

Clicking a line on the map highlights all routes from both endpoints, which lets you see what each airport specializes in. Airline industry coverage often talks about "gateways", and this lets you see them firsthand.

New York JFK is a gateway east. Dense connections deep into Europe, but comparatively fewer to Latin America and Asia.

Los Angeles is a gateway west. Much better connections to Australia, Asia, and Hawaii, with thinner lines to European second cities.

Gateways north and south

Using the word "gateway" is more common with airlines as they talk about their hubs away from the top world cities like NYC and LA.

Miami is a gateway south. American Airlines calls Miami "its primary gateway to Latin America and the Caribbean." The map confirms it: connections to Latin American second cities that even JFK doesn't have, fewer lines to Europe, nothing to Asia.

Seattle is a gateway north. The airport brands itself as "The Gateway to Alaska," and sure enough, Seattle–Anchorage is a top-50 route. (Delta also built Seattle into an Asia gateway over the past decade, and you can see those routes too, though they're still much smaller than LAX's.)

Conclusion

I made this map because I was stuck at home and missed the world. I ended up staring at it for hours not to plan any trip, just to watch the lines and think about where they go.

There's a kind of knowledge that only comes from looking. I could have told you before I started that JFK was a European gateway and LAX was a Pacific one, but it's different to see it.

Appendix: Making the map

The hard part of this was finding the dataset.

The underlying data comes from the Bureau of Transportation Statistics T-100 Segment (All Carriers).

The key fields are:

origin airport (ORIGIN)
destination airport (DEST)
seats (SEATS)

Then I essentially ran a SQL query to get seats between airports...

select ORIGIN, DEST, sum(SEATS)
from t_100_table
where ORIGIN < DEST -- only get one direction of the flights 
group by ORIGIN, DEST 
order by 3 desc

...joined that result with the coordinates of different airports, and turned it into a line for plotting in JavaScript.

I then limited it to only show the routes that have as many seats as between Chicago ORD and the small commercial airport closest to where I grew up.

Blog Title

Designing an Architecture for StreetSmart

Context: What we built

Decision 1: Use tech that the team knows

Decision 2: Maintain a $0/yr annual budget

Decision 3: Don't blindly trust the LLM

First model: panoptic mask2former

Next: Gemini LLM

Three: Apply rules to stick it together

Evaluating this structure

In conclusion

Nine Months of Unplanned PG&E Electricity Outages

Data collection and transformation

Step 0: CalOES gathers information

Step 1: gather data

Step 2: Compress data

Step 3: Ingest data

Step 4: Select data

Step 5: Compute polygons with the same outage performance

Step 6: Turn the outage observations into hours

Step 7: Simplify the shapes and put it on a map

Sources of error

CalOES data does not match what's happening on the ground

Short outages are badly represented

CalOES does not update

Downloaded invalid JSON

My data collection infrastructure fails

Outages caused by privately owned infrastructure

Validation

Validating one outage at my house

PG&E text messages

PG&E hourly power usage

Data downloaded from CalOES

And on the map

Validating months of outages at my house

Validating with exported power usage

Showing the outline of PG&E's service area

PG&E outages outside PG&E territory

Optimizing the basemap

Copyright information

Sunset Here is Sunrise Where?

Eyeballing an answer

Getting places around the world

Finding places that mean something to someone

Building with Claude

Conclusion

Finding the City Based On What's Open Now

Similar Work: Areas of Interest in Google Maps

Using Open Establishments to Figure Out Where People Are

Comparing Open Areas to Google Maps

How I Built It (the fun part)

One: Get the Data

Two: Find what's open

Three: Visualize the social gravity

Four: Cluster overlapping circles

Four: Isolate areas where 3+ circles overlap

Five: Scale across time

How I Built It (the technical tricks)

Load OSM data in to PostGIS

Interpreting OSM open hours

Working with data in PostGIS

Sharing my results with the world

Reclaiming My Media Diet with RRSS - A Custom RSS Reader

Why I love Hacker News

What to read instead

Designing a replacement

User interface goals

Getting the news via RSS

Designing and Building the RRSS Application

Deploying RRSS on a Free PHP Host

Run it yourself!

Appendix: Finding RSS feeds

Appendix: Why I dislike the alternatives

Newsletters are hard to skim

Newsletters arrive on the wrong cadence

Too many publisher apps

Other RSS readers don't work for me

Social Media is miserable

Read a physical newspaper

Power Outages in SF Bay Area (Part 1)