tags:
- “#project-ideas"
- "#in-progress”
- machine-learning
Status
Temporarily unresolvable data curation difficulties
Idea
This is a fuzzy video generation idea where we ultimately generate gameplay clips for starcraft 2 games. This essentially means the model will only be trained and used for Starcraft gameplay videos.
Current Problems
- sc2 tournament replays on YouTube have UI, we don’t necessarily want UI
- if we don’t want UI, we need to either get our hands on the original replay files and pass in sc2 engine, or we cut parts of the screen to omit UI.
The problem with the second (cutting UI out of the video) approach is that we are leaving out data: data that are actually part of the game; this is not like trimming the empty spaces around an MNIST image, this is like if we have a “6” but only cut the two sides so much it becomes Ξ.
- what’s the takeaway from this? We can first think of what data we need for such multi-modal model:
- audios by the casters for a game;
- replay video with camera movements;
- the second part of the data is critical, because it is not just what we see, but also what the casters see: you can probably tell how we can intuitively train them together. This presents a significant data curation challenge:
- we either preserve the UI for now (most viable, but the most compromised, too), or:
- we literally collect all raw replays files for the past notable tournaments, AND THEN we replicate the camera movements as all the casted games.
Can an AGI do the latter please… damn