Skip to content
BranchesTags

Repository files navigation

BuildStock Processor

A Python package for downloading and analyzing NLR ComStock and ResStock metadata, annual results, upgrade packages, and individual-building time series from the public OEDI data lake.

The package exposes two processors:

  • ComStockProcessor for commercial whole-building records.
  • ResStockProcessor for residential dwelling-unit records.

Both use shared download, cache, filtering, upgrade-lookup, and time-series infrastructure. They remain separate because their releases, metadata partitioning, building types, crosswalk formats, and record semantics differ.

Installation

Install uv and sync the project's dependencies:

pip install uv
uv sync --group dev

The project uses a standard src/ package layout. uv sync installs buildstock_processor into the project environment as an editable package.

Local Development

The repo also includes a FastAPI backend (api/) and an Angular webapp (webapp/) for interactively exploring BuildStock data in a browser. Install the extra dependencies, then run both together with:

uv sync --group dev --group api
cd webapp && npm install && cd ..

make dev

make dev runs dev.sh, which starts the FastAPI backend at http://localhost:8000 and the Angular frontend at http://localhost:4200, and stops both on Ctrl+C.

Documentation

Quick Start

from pathlib import Path

from buildstock_processor import ComStockProcessor, ResStockProcessor

comstock_dir = Path("datasets/comstock")
comstock_dir.mkdir(parents=True, exist_ok=True)

offices = ComStockProcessor(
    state="DC",
    county_name="All",
    building_type="SmallOffice",
    upgrade="0",
    base_dir=comstock_dir,
).process_metadata(save_dir=comstock_dir)

resstock_dir = Path("datasets/resstock")
resstock_dir.mkdir(parents=True, exist_ok=True)

multifamily_units = ResStockProcessor(
    state="DC",
    county_name="All",
    building_type="Multi-Family with 5+ Units",
    upgrade="0",
    base_dir=resstock_dir,
).process_metadata(save_dir=resstock_dir)

ComStock rows represent commercial buildings. ResStock rows represent independently sampled dwelling units, including units in multifamily buildings; they are not whole-building records. Read Data model and limitations before combining or weighting the two stocks.

Data Dictionary

The package includes a parseable data dictionary in docs/data_dictionary.md and src/buildstock_processor/data_dictionary.json. It covers building types, annual result variables with parsed units, and release-specific measure upgrade packages. The same data is available without downloads from the processor classes:

from buildstock_processor import BuildStock, ComStockProcessor, ResStockProcessor

BuildStock.building_types
ComStockProcessor.building_types
ComStockProcessor.data_dictionary.result_variables_by_unit("kwh")
ComStockProcessor.data_dictionary.upgrade_packages("release_3")

ResStockProcessor.building_types
ResStockProcessor.data_dictionary.result_variable_names

Upgrade package ids are release-specific, so they are grouped by release in both the Markdown/JSON dictionary and the Python API.

ENERGY STAR Building Type Crosswalk

The package also includes a best-effort crosswalk from ENERGY STAR Portfolio Manager property types to BuildStock building types, in docs/energy_star_crosswalk.md and src/buildstock_processor/energy_star_crosswalk.json. ENERGY STAR's ~84 property types are far more granular (and organized differently) than BuildStock's 15 ComStock/5 ResStock building types, so several ENERGY STAR types have no close BuildStock equivalent (e.g. "Zoo", "Swimming Pool", open-air stadiums, parking structures). This is not an official NLR/EPA publication -- every entry records a match_quality ("exact", "approximate", or "unmapped") and notes explaining the reasoning, so callers can judge whether an approximate match is good enough for their use case.

from buildstock_processor import map_energy_star_property_type, energy_star_property_types_for_buildstock_type

mapping = map_energy_star_property_type("Bank Branch")
# EnergyStarMapping(energy_star_property_type='Bank Branch', buildstock_product='comstock',
#                    buildstock_building_type='SmallOffice', match_quality='approximate', notes='...')

energy_star_property_types_for_buildstock_type("comstock", "SmallOffice")
# ('Bank Branch', 'Financial Office', 'Fire Station', ...) -- reverse lookup

Composite ("Mixed-Use") Building Types

Real buildings are often not well represented by a single BuildStock building type -- e.g. a building that is 70% office space over 30% ground-floor retail. CompositeBuildingType (see docs/usage.md and 03_composite_building_example.ipynb) models this as a fraction-weighted combination of two or more (product, building_type) components -- including mixing ComStock and ResStock components (e.g. ground-floor retail under apartments). pull_composite_time_series() downloads a representative building's time series per component and auto-stitches them into one synthetic composite time series; combine_composite_time_series() does the combining step alone if you've already downloaded component time series yourself.

from buildstock_processor import CompositeBuildingType, pull_composite_time_series

office_retail = CompositeBuildingType.from_fractions(
    "70% MediumOffice / 30% RetailStripmall",
    {("comstock", "MediumOffice"): 0.7, ("comstock", "RetailStripmall"): 0.3},
)
combined, components = pull_composite_time_series(office_retail, save_dir=composite_dir, state="DE")

Package Layout

src/buildstock_processor/
|-- __init__.py                  # public processors and abstract extension types
|-- _base.py                     # BuildStockProcessor ABC and shared workflows
|-- comstock.py                  # ComStock implementation and releases
|-- resstock.py                  # ResStock implementation and releases
|-- data_dictionary.py           # packaged building-type/result-variable/upgrade-package dictionary
|-- energy_star_crosswalk.py     # packaged ENERGY STAR -> BuildStock building-type crosswalk
`-- composite.py                 # CompositeBuildingType + combine/pull composite time series

ComStockProcessor Class

The ComStockProcessor class is located in src/buildstock_processor/comstock.py and provides methods to download and process ComStock building data.

Initialization

from pathlib import Path
from buildstock_processor import ComStockProcessor

# Initialize the processor
processor = ComStockProcessor(
    state="CA",           # 2-letter state abbreviation
    county_name="All",    # County name, a list of county names, or "All" (see "Searching for Buildings" below)
    building_type="All",  # Building type or "All"
    upgrade="0",          # Upgrade identifier (0 = baseline)
    base_dir=Path("./datasets/comstock"),  # Local directory to save data
    release="release_3",  # Optional: which ComStock release to use (see "Supported Releases" below)
    min_sqft=None,        # Optional: only include buildings at least this large
    max_sqft=None,        # Optional: only include buildings at most this large
)

Supported Releases

ComStock is periodically republished with updated building samples, results, and file layouts. ComStockProcessor supports the last three published releases of the ComStock AMY2018 dataset, selected via the release argument:

release value Description
"release_1" ComStock AMY2018 Release 1
"release_2" ComStock AMY2018 Release 2
"release_3" ComStock AMY2018 Release 3 (default)

If release is omitted, the most recent supported release is used. Passing an unsupported value raises a ValueError listing the currently supported releases. The full set of supported releases and their on-disk locations are defined in SUPPORTED_RELEASES in src/buildstock_processor/comstock.py — when NLR publishes a new release, add it there and drop the oldest entry to keep a rolling window of three supported releases.

OEDI year and release paths

Both processors use the same OEDI path convention: the publication year is the directory immediately below the building-stock prefix, and the dataset folder contains the weather year and release number. The currently supported paths are:

Dataset OEDI path suffix
ComStock AMY2018 Release 1 2025/comstock_amy2018_release_1/
ComStock AMY2018 Release 2 2025/comstock_amy2018_release_2/
ComStock AMY2018 Release 3 2025/comstock_amy2018_release_3/
ResStock AMY2018 Release 1 2025/resstock_amy2018_release_1/
ResStock AMY2012 Release 1 2025/resstock_amy2012_release_1/

The release argument remains the release identifier (release_1, release_2, or release_3); it is not the publication year. ResStock's weather_year selects the AMY2018 or AMY2012 dataset within the supported 2025 release.

Methods

process_metadata(save_dir: Path) -> pd.DataFrame

Downloads and processes ComStock metadata with filtering based on the class constraints.

  • ComStock metadata is published per state/county/upgrade partition (not as a single national file), so this discovers the relevant partitions for the requested state (or every available state, if state="All") and downloads them in parallel
  • Filters by county and building type as specified during initialization
  • Saves filtered results as a CSV file (namespaced by release, so different releases don't collide)
  • Returns a pandas DataFrame with the filtered metadata

Note: Because metadata is only partitioned by state and county (not building type), requesting a specific county_name does not reduce how many files are downloaded, and state="All" downloads every state's and county's partition files, which can be a large number of downloads.

process_building_time_series(data_frame, save_dir: Path) -> tuple

Downloads time series data for buildings specified in the input DataFrame using parallel execution.

  • Uses multi-threading to download building time series files efficiently
  • Skips downloading files that already exist locally
  • Downloads from the ComStock AWS S3 bucket
  • Returns paths and building IDs of downloaded files

Searching for Buildings, Then Downloading Their Time Series

process_metadata()'s constraints (county_name, building_type, min_sqft/max_sqft) act as a search: find the buildings matching some criteria, then pass the resulting DataFrame straight to process_building_time_series() to download time series data only for those buildings.

  • county_name accepts a single county, "All", or a list of counties, which is useful for querying a metro area that spans several counties (a single state/county partition can't represent that on its own).
  • min_sqft/max_sqft filter by building square footage (in.sqft..ft2).
# All office buildings under 10,000 sqft in the Denver metro area
processor = ComStockProcessor(
    state="CO",
    county_name=["Denver County", "Arapahoe County", "Jefferson County", "Adams County", "Douglas County", "Broomfield County"],
    building_type="SmallOffice",
    upgrade="0",
    base_dir=base_dir,
    max_sqft=10_000,
)
matching_buildings = processor.process_metadata(save_dir=base_dir)

# Download time series data only for the buildings that matched
timeseries_dir = base_dir / "timeseries"
timeseries_dir.mkdir(exist_ok=True)
paths, building_ids = processor.process_building_time_series(matching_buildings, save_dir=timeseries_dir)

Note: Since metadata is only partitioned by state and county (not building type or square footage), min_sqft/max_sqft and requesting specific counties don't reduce how many partition files are downloaded -- they're applied locally, after downloading, the same way building_type already is. Cache filenames (the ..._selected_metadata.csv files) encode all of these filters, so different searches against the same state/upgrade don't collide with each other's cached results.

Comparing Buildings Across Measure Packages

Each ComStock "upgrade" represents a different energy-efficiency measure package (e.g. a heat pump RTU, VRF system, or envelope upgrade) applied to the same baseline building sample. ComStockProcessor provides methods to discover those packages and download metadata for several of them at once, so you can compare how a building performs under different packages.

list_upgrades(save_dir: Path) -> dict[str, str]

Downloads (and caches) the release's upgrades_lookup.json, returning a mapping of upgrade id -> package name, e.g. {"0": "Baseline", "1": "Variable Speed HP RTU, Electric Backup", ...}. Which upgrade ids exist, and what they mean, is different for every release (release_1, release_2, and release_3 each have a different number of packages and, in some cases, different ids for what looks like the same package).

process_metadata_for_upgrades(save_dir: Path, upgrades: list[str] | None = None) -> pd.DataFrame

Downloads and combines metadata for multiple upgrades into a single DataFrame (reusing the same per-upgrade download/caching as process_metadata()). Every row already has an upgrade id column and an in.upgrade_name column, so you can group by bldg_id to compare a building's results (energy consumption, savings, etc.) across packages. If upgrades is omitted, every upgrade available for the release is downloaded and combined.

# Compare Delaware small offices under the baseline vs. a heat pump RTU package
processor = ComStockProcessor(
    state="DE", county_name="All", building_type="SmallOffice", upgrade="0", base_dir=base_dir
)
combined_df = processor.process_metadata_for_upgrades(save_dir=base_dir, upgrades=["0", "1"])

# One row per building per package, ready to compare
combined_df.groupby("bldg_id").apply(
    lambda g: g.set_index("upgrade")["out.site_energy.total.energy_consumption..kwh"]
)

Note: A building can appear more than once per upgrade in the "full" metadata (it may be reused to represent multiple census tracts, each with its own weight). If you only need each building's simulated performance, group/filter by bldg_id and upgrade and take the first row of each group.

Comparing measure packages across releases

Because upgrade ids aren't stable between releases, ComStock also publishes a measure_name_crosswalk.csv that maps a stable measure_id (e.g. "hvac_0005") to the upgrade id/name used for that measure in each release.

  • get_measure_crosswalk(save_dir: Path) -> pd.DataFrame — downloads (and caches) the crosswalk table for the configured release. A release's crosswalk only covers itself and earlier releases, so release_3 (the newest) has the most complete crosswalk, covering all three currently-supported releases.
  • find_upgrade_id(save_dir: Path, measure_id: str, target_release: str | None = None) -> str | None — looks up the upgrade id for a stable measure_id in a specific release (defaults to the processor's own release). Returns None if that measure wasn't included in the target release, and raises a ValueError if the target release isn't covered by the currently loaded crosswalk.
processor = ComStockProcessor(
    state="DE", county_name="All", building_type="All", upgrade="0", base_dir=base_dir, release="release_3"
)

# "Heat Pump RTU" happens to be upgrade "1" in every currently-supported release, but that's not
# guaranteed for every measure -- use find_upgrade_id() rather than hardcoding ids across releases.
upgrade_id_r3 = processor.find_upgrade_id(save_dir=base_dir, measure_id="hvac_0005")               # "1"
upgrade_id_r1 = processor.find_upgrade_id(save_dir=base_dir, measure_id="hvac_0005", target_release="release_1")  # "1"

Usage Example

from pathlib import Path
from buildstock_processor import ComStockProcessor

# Set up directories
base_dir = Path("./datasets/comstock")
timeseries_dir = base_dir / "timeseries"
for d in [base_dir, timeseries_dir]:
    d.mkdir(parents=True, exist_ok=True)

# Initialize processor for California data
processor = ComStockProcessor(
    state="CA",
    county_name="All",
    building_type="All",
    upgrade="0",
    base_dir=base_dir,
)

# Download and filter metadata
metadata_df = processor.process_metadata(save_dir=base_dir)

# Download time series data for buildings in metadata
paths, building_ids = processor.process_building_time_series(
    metadata_df,
    save_dir=timeseries_dir
)

Data Source

The processor downloads data from the ComStock dataset hosted on AWS S3. For example, the default release:

  • Base URL: https://oedi-data-lake.s3.amazonaws.com/nrel-pds-building-stock/end-use-load-profiles-for-us-building-stock/2025/comstock_amy2018_release_3/
  • Data Explorer: OpenEI Data Lake Explorer

Performance Features

  • Parallel Downloads: Uses ThreadPoolExecutor for concurrent file downloads
  • Smart Caching: Skips downloading files that already exist locally
  • Progress Tracking: Shows download progress with tqdm progress bars
  • Efficient Filtering: Uses pandas parquet filtering for large datasets

ResStockProcessor Class

The ResStockProcessor class (in src/buildstock_processor/resstock.py) provides the same interface for NLR's residential building stock dataset, ResStock, which is hosted on the same OEDI data lake. It shares its download/caching/upgrade-lookup infrastructure with ComStockProcessor via a common abstract BuildStockProcessor base class (src/buildstock_processor/_base.py), but has its own metadata layout and release registry, since ResStock's file structure and building-type categories differ from ComStock's.

from pathlib import Path
from buildstock_processor import ResStockProcessor

processor = ResStockProcessor(
    state="CA",
    county_name="All",
    building_type="Multi-Family with 5+ Units",  # see "Handling Multifamily Buildings" below
    upgrade="0",
    base_dir=Path("./datasets/resstock"),
    release="release_1",
    weather_year="amy2018",  # Optional; "amy2012" is also supported
)
metadata_df = processor.process_metadata(save_dir=processor.base_dir)

Key differences from ComStockProcessor

  • Metadata partitioning: ResStock metadata is partitioned only by state, not by state and county like ComStock. Specifying county_name doesn't reduce how much is downloaded (there's only one file per state/upgrade); it's filtered locally afterward.
  • county_name format: ResStock's in.county_name values don't include the state prefix ComStock uses -- pass "Kent County", not "DE, Kent County". Like ComStock, county_name also accepts a list of counties for metro-area-style searches, and min_sqft/max_sqft filter by dwelling unit square footage -- see "Searching for Buildings, Then Downloading Their Time Series" above.
  • building_type values: use one of RESSTOCK_BUILDING_TYPES (ResStock's residential housing categories), not ComStock's commercial building types:
    • Mobile Home, Single-Family Detached, Single-Family Attached, Multi-Family with 2 - 4 Units, Multi-Family with 5+ Units
  • Releases and weather years: "release_1" is the current supported 2025 ResStock release. Use weather_year="amy2018" (the default) or weather_year="amy2012" to select the corresponding 2025 OEDI dataset.
  • Measure crosswalk format: for releases that publish a measure crosswalk, get_measure_crosswalk() downloads an Excel file, not a csv like ComStock (this is why openpyxl is a dependency). The 2025 AMY2012 dataset does not publish a separate measure crosswalk in OEDI; use list_upgrades() for its release- and weather-specific upgrade package ids.

Handling Multifamily Buildings

ResStock simulates individual dwelling units, not whole buildings: a single "Multi-Family with 5+ Units" row is one apartment unit, not the building it's in. There's no shared "building id" tying multiple sampled units back to the same real building -- each unit is an independently sampled, weighted record. Relevant columns:

  • in.geometry_building_type_recs — the housing type (one of RESSTOCK_BUILDING_TYPES); filter/group on this to select multifamily units.
  • in.geometry_building_number_units_mf — how many units are in that unit's (whole) building.
  • in.geometry_building_horizontal_location_mf / in.geometry_building_level_mf — the unit's position within the building (corner/middle, top/bottom floor), which affects heat transfer through shared walls/floors/ceilings with neighboring units.
  • weight / in.units_represented — the sampling weight used to scale a simulated unit up to the full housing stock population.
processor = ResStockProcessor(
    state="DE", county_name="All", building_type="Multi-Family with 5+ Units", upgrade="0", base_dir=base_dir
)
multifamily_units_df = processor.process_metadata(save_dir=base_dir)

# Weighted total number of real housing units this sample represents
multifamily_units_df["weight"].sum()

process_metadata_for_upgrades, list_upgrades, get_measure_crosswalk, find_upgrade_id

ResStockProcessor supports process_metadata_for_upgrades and list_upgrades for every supported release. get_measure_crosswalk and find_upgrade_id are available only for ResStock releases that publish a measure crosswalk; currently that is 2025 release_1 with weather_year="amy2018".

Development

The processor includes comprehensive unit and integration tests validating both ComStockProcessor (tests/test_comstock_processor.py) and ResStockProcessor (tests/test_resstock_processor.py).

Running Tests

Run specific test categories:

# Unit tests only (fast)
uv run pytest tests/ -m "unit" -v

# Integration tests (downloads small datasets)
uv run pytest tests/ -m "integration" -v

# All tests including large dataset downloads
TEST_DATA=true uv run pytest tests/ -m "integration" -v

# Run all tests
uv run pytest tests -v

Test Categories

  • Unit tests: Fast tests that verify initialization and basic functionality
  • Integration tests: Tests that download and process real ComStock and ResStock data. test_all_state_filter mocks state discovery down to a couple of small states so it can exercise the real state="All" code path without downloading every state's/county's metadata partition.

Execute the notebooks as integration checks:

uv run jupyter nbconvert --to html --execute 01_data_sampling_example.ipynb
uv run jupyter nbconvert --to html --execute 02_washington_dc_stock_analysis.ipynb

Committing

Before pushing changes to GitHub, run pre-commit to format the code consistently. pre-commit is installed as part of the dev dependency group, so run it via uv:

uv run pre-commit run --all-files

If this doesn't work, try:

uv sync --group dev
uv run pre-commit run --all-files

About

No description, website, or topics provided.

Resources

Readme
License
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages