Couldn’t we have a lead box lined with these radiation to electricity converters with a small amount of radioactive material in the center, and have an energy generating device that would last for thousands or even millions of years? Imagine putting the sun in a box lined with solar cells, but on a much smaller scale.
Is there a reason this wouldn’t work?
Yes and we already have and use them. Albeit only on NASA probes leaving earth.
Both of the curiosity rovers use this as their power plant. A bunch of plutonium sits in that little tail like stub on them and generates perpetual power for them.
To add to your answer, the reason they're not used on Earth, besides the radioactivity dangers, is that they just can't produce very much power.
Would it be a way to reuse nuclear waste?
The recycling process that France currently uses recovers about 96% of the radioactive material remaining in nuclear waste and produces new fuel rods usable in conventional reactors. Spent fuel rods from conventional reactors can also be used in other types of reactors. For whatever reason, we in the US just shove all of our nuclear waste in old mines like it's still 1950 but technology has continued to develop and places that are more invested in nuclear energy have much better ways of dealing with the waste.