Have you tried keeping your windows all the same size (including the edge windows) and specifying `boundless=True` and a `fill_value=some appropriate value` to fill out your windows in your ` `dataset.read` call?

https://rasterio.readthedocs.io/en/latest/api/rasterio.io.html#rasterio.io.DatasetReader.read

i.e. (untested)

dataset.read(window=window, boundless=True, fill_value=dataset.nodata)