CIS 565: GPU Programming and Architecture
University of Pennsylvania
Fall 2019
Final Projects Overview Presentation
The final project gives you an opportunity to embark on a large GPU programming endeavor of your choice. You are free to select an area in graphics, GPU computing, or both.
You can
- fulfill a real need in the community,
- explore a new API,
- reproduce the results of recent research,
- add a novel extension to existing work, or
- implement something completely original.
Expect this to be 2-3x more work than the hardest project this semester. The best projects will require a time commitment of at least 100-125 hours per student. It will be worth it.
- Form teams of two or three students. Each team member will receive the same grade. Teams of four will be considered, but will be expected to build something truly amazing like this.
- Teams of one are discouraged and will only be considered on a case-by-case basis.
- Use GitHub. We encourage, but do not require, you to develop in a public repo as open source.
- Programming language, graphics and compute APIs, and target platforms are up to you.
- You are allowed to use existing code and libraries. These should be prominently credited and your code should be easily identifiable. Be incredibly clear about what is your work and what is not.
- Add custom de-noising algorithms to Path Tracer and/or DXR
- Used everywhere real-time ray/path-tracing is used
- Buzz in all of rendering community
- Can put you in touch with Alain Galvan (Nov 20 guest speaker)
- Blog post: https://alain.xyz/blog/raytracing-denoising
- Real-time rendering pipeline + add ons
Resources:
- http://www.realtimerendering.com/erich/Ray%20Tracing%20at%20Last.pptx
- https://github.com/NVIDIAGameWorks/DxrTutorials
- https://github.com/Microsoft/DirectX-Graphics-Samples/tree/master/Samples/Desktop/D3D12Raytracing
- https://developer.nvidia.com/rtx/raytracing/dxr/DX12-Raytracing-tutorial-Part-1
- https://developer.nvidia.com/rtx/raytracing/dxr/DX12-Raytracing-tutorial-Part-2
- http://intro-to-dxr.cwyman.org
- http://cwyman.org/code/dxrTutors/dxr_tutors.md.html
The field is open and vast. You can suggest everything from improving computer vision to page rank and so on. Pretty much everything is on the table for this.
- Huge amounts of open Photogrammetry and Lidar models available today
- Big effort going on in the US Army - Look up “One World Terrain”
- Watch: USC ICT Videos
- https://www.spiedigitallibrary.org/conference-proceedings-of-spie/10207/1/Visualizing-UAS-collected-imagery-using-augmented-reality/10.1117/12.2262864.short?SSO=1
- https://arxiv.org/ftp/arxiv/papers/1711/1711.10693.pdf
- http://trajectorymagazine.com/rapid-terrain-generation/
- https://scholarspace.manoa.hawaii.edu/bitstream/10125/59634/1/0194.pdf
- http://ict.usc.edu/pubs/Producing%20Usable%20Simulation%20Terrain%20Data%20from%20UAS-Collected%20Imagery.pdf
Note: Shehzan can help with source data.
Background
- Graphics Topics in Chrome
- Designing a Next-Gen Graphics API for the Web by Kai Ninomiya
- https://github.com/austinEng/webgpu-samples
- https://doc.babylonjs.com/extensions/webgpu
- https://webkit.org/demos/webgpu/
- https://github.com/gpuweb/gpuweb/wiki/Implementation-Status
Ideas
- glTF 2.0 PBR renderer for WebGPU
- Vulkan backend for WebGPU
- CUDA rasterizer backend for WebGPU
- Tensorflow operators using WebGPU compute shaders
- Anything to help with WebGPU design
- Prototype a graphics engine in C++/OpenGL ES and "compile" it for the web using WebAssembly. Provide benchmarks.
- Answer the question: how can an existing JavaScript codebase use WebAssembly? Rewrite the whole thing in C++? Rewrite hot fine-grained modules, e.g., matrix and vector types? Rewrite hot coarse-grained modules, e.g., the physics engine?
Also talk to Austin Eng, who used WebAssembly for his Senior Design, and now is a part of the Chrome team. Austin will also be a guest speaker on 11/26.
glTF Working Group is working to introduce a fully physically based material model. Would be amazing to integrate this with other engines as a prototype. I can introduce you to the people at the forefront of the specification and the Working Group.
Collaborate with Cesium on creating a 3D Tiles runtime for Unity. Leverage the glTF importer to bring in massive mesh and point cloud datasets into Unity.
Disclosure: Shehzan works for Cesium.
https://github.com/gpujs/gpu.js
Build something with gpu.js:
gpu.js is a JavaScript Acceleration library for GPGPU (General purpose computing on GPUs) in Javascript. gpu.js will automatically compile simple JavaScript functions into shader language and run them on the GPU.
3D Mapping WebGL Engine
Disclosure: Shehzan works for Cesium.
Expand on https://cesium.com/blog/2017/10/23/integrating-cesium-with-threejs/ to fully explore integrating Cesium with other WebGL engines such as three.js and BabylonJS - both by overlaying two canvases and by combining the rendering loop.
Improve the quality of the global-scale ocean/water rendering in Cesium. Add any required WebGL 2 features to Cesium.
Resources
- https://www.fxguide.com/featured/assassins-creed-iii-the-tech-behind-or-beneath-the-action/
- http://the-witness.net/news/2012/08/regarding-water/
- http://vterrain.org/Water/
Suggested features from BabylonJS Team:
- Cascaded Shadows
- Area Lights
- Real time environment filtering
Contribute to renderdoc.
Does batching still help with Vulkan? If so, when it is worth it?
Create a domain-specific shading language targeting SPIR-V. Perhaps a language tuned for CSG, voxels, or ray marching.
Implement a new abstraction layer using Vulkan that is higher level than Vulkan, but lower level than, for example, Unity.
Prototype a small engine with multithreading for LOD and culling.
Write a Vulkan for OpenGL developers tutorial series.
- Performance analysis of porting an engine to VR
- Naive stereo rendering vs. multi-view
- Implement barrel distortion and chromatic aberration filters with optimizations
- In practice, use filters from VR vendors to avoid making users sick
- More precise object culling with a volume derived from the hidden area mesh
- Is this practical and useful?
- In each eye, most of the same fragments will pass the depth test, can this coherence be used to optimize?
- Evaluate the checkerboard stencil in radial density masking as a general optimization
- Does it have better visual quality than rendering a scene at half-resolution and upsampling with only a nominal cost?
- Expand radial density masking / fixed foveated rendering to apply geometric/material LOD to objects in the periphery
- Implement a simulation for avoiding judder
- Compare async timewarp, spacewarp, and late orientation warping, and heuristics for selecting between them
- Improve spacewarp with a sparse voxel octree
- Not sure if this will work, but perhaps could improve occlusion artifacts at a reasonable performance and memory cost
3D Machine Learning = Computer Vision + Graphics + Machine Learning
- http://graphics.stanford.edu/courses/cs468-17-spring/
- https://github.com/timzhang642/3D-Machine-Learning
GPU accelerate parts of autonomous cars.
- https://github.com/carla-simulator/carla
- https://www.vectorzero.io/
- https://7dlabs.com/synscapes-overview
- https://www.vectorzero.io/products
Use Jetson kits for interesting applications in computer vision, autonomous vehicles, drones, robotics etc.
See success stories.
Ps. You may need to procure hardware, but we can help with references.
Please do not propose yet-another-CUDA-fluid/smoke/cloth simulation unless it has a unique spin such as using
- Vulkan compute shader
- NXT compute shader
- Multi-GPU
- A very sophisticated simulation
Likewise, please do not propose extend-my-CUDA-path-tracer-with-these-n-effects unless it is truly unique.
For inspiration, browse the CIS 565 final projects from previous semesters:
A guideline is that your project should be better than last semester's projects; that is how we move the field forward.
- Fall 2018
- Fall 2017
- Pollux: Metal Pathtracer by Youssef Victor and Will Ho
- Organic Mesh Growth with SDFs by Mariano Merchante and Mohamad Moneimne
- BabylonJS Post-Processing and Weather System by Hanming Zhang and Linshen Xiao
- Fall 2016
- Vulkan PBR Renderer by Jian Ru
- Vulkan Forward+ Renderer by Ruoyu Fan and Xueyin Wan
- PIC/FLIP Fluid with WebGL by Austin Eng
- Fall 2015
- Forward+ Renderer using OpenGL/Compute Shaders by Bradley Crusco and Megan Moore
- WebGL Fragment Shader Profiler by Sally Kong and Terry Sun
- GPU Cloth with OpenGL Compute Shaders by Gary Li
- Fall 2014
- Bidirectional Path Tracer in CUDA by Paula Huelin Merino and Robbie Cassidy
- GPU-Accelerated Dynamic Fracture in the Browser with WebCL by Kai Ninomiya and Jiatong He
- Uniform grid and kd-tree in CUDA by Jeremy Newlin and Danny Rerucha
- Fall 2013
- Surface Mesh Reconstruction from RGBD Images by Collin Boots and Dalton Banks
- Sparse Voxel Octree by Cheng-Tso Lin
- Terrain tessellation by Mikey Chen and Vimanyu Jain
- GPU Photon Mapper by Ishaan Singh, Yingting Xiao, and Xiaoyan Zhu
- Fall 2012
- Non-photorealistic Rendering by Kong Ma
- Procedural Terrain by Tiju Thomas
- KD Trees on the GPU by Zakiuddin Shehzan Mohammed
- Spring 2012
- Single Pass Order Independent Transparency by Sean Lilley
- GPU-Accelerated Logo Detection by Yu Luo
- GPU-Accelerated Simplified General Perturbation No. 4 (SGP4) Model by Matthew Ahn
- Spring 2011
- Fast Pedestrian Recognition on the GPU by Fan Deng
- Screen Space Fluid Rendering by Terry Kaleas
- Deferred Shader with Screen Space Classification by Sean Thomas
Browse these for ideas galore!
- Journal of Computer Graphics Techniques
- Advances in Real-Time Rendering SIGGRAPH courses
- Ke-Sen Huang's conference pages - papers from SIGGRAPH, Eurographics, I3D, and elsewhere
- Real-Time Rendering Portal and WebGL Resources - links to an amazing amount of content
Also look at the course website for more resources.
Sign up for a time slot ASAP. Only team member names are needed to sign up.
Your project pitch is a 15-minute meeting with Shehzan, and the TAs, and a write-up no longer than one page that includes an overview of your approach with specific goals and weekly milestones for 11/18, 11/25, 12/02, and 12/9. First, focus on why there is a need for your project. Then describe what exactly you are going to do. In addition to your write-up, provide supplemental figures, images, or videos.
Think of your pitch as if you are trying to get a startup funded, convincing your company to start a new project, or responding to a grant. For an example, see Forward+ Renderer using OpenGL/Compute Shaders by Bradley Crusco and Megan Moore (this example pitch does not include weekly milestones, which is new this semester).
Before 11/4 5:00pm:
- Make a Piazza post with your one page pitch (attach a PDF) and any supplemental material to Instructors by 5:00pm on Monday 11/04. Sooner is better. To really hit a home run, consider prototyping something before the meeting.
- Use the subject as
[CIS 565 2019 Final Project] Name1, Name2, Name3. Include a project/team name, if chosen.
- Use the subject as
You are free to start working on your project before the pitch. As long as you have followed the guidelines, it is more than likely that your project will be approved. We may ask you to tweak it, move the milestones etc.
After the meeting:
- Push your pitch to a new GitHub repo for your project.
- Add the names of the team members, a project name (if any) and the link to the Github repository (if open source) to the List of Final Projects.
On or before 11/10:
- Choose your shadow team.
- Shadow teams must be used for practicing presentations and good for getting presentation content/timing into shape. Shadow teams may be used for bouncing technical ideas.
- Plan to meet once weekly for 30-60 minutes
- Ideally, your shadow team should be working on a similar idea for their final project
Your first presentation should be 4 minutes. Present your work-in-progress. Your presentation should include a few slides, plus videos, screenshots, or demos if possible. Be sure to
- Demonstrate working code (videos and screenshots are OK; it doesn’t have to be live).
- Provide a roadmap with future weekly milestones up to and including the final presentation on Monday, December 9.
See the Cesium Presenter's Guide for tips on presenting. Be sure to present as a team; for a great example, see http://www.youtube.com/watch?v=OTCuYzAw31Y
After class, push your presentation to your GitHub repo.
A 5 minute presentation on your progress over the past week. Demonstrate how you reached or exceeded the goals for this milestone. If you didn't reach them, explain why. Remind us of your upcoming milestones, and adjust them if needed.
After class, push your presentation to your GitHub repo.
Same format as Milestone 2.
Follow the same guidelines as all the regular class projects. At this point do not include your milestone information in the readme. You may use other readmes or PDFs for milestone progress.
Make sure to include your project pitch, milestone PDFs, images, videos etc in a directory.
Be sure to include:
- Final code - should be clean, documented, and tested
- A detailed README.md including:
- Name of your project
- Your names and links to your website/LinkedIn/twitter/whatever
- Choice screenshots including debug views
- Link to demo if possible. WebGL demos should include your names and a link back to your github repo
- Overview of technique and links to references
- Link to video
- Detailed performance analysis
- Install and build instructions
Note: Presentations should be submitted by 5pm on Monday 12/9.
5-minute final presentation.
This should be a polished presentation with a 1-2 slide overview of the technical approach at a high-level of abstraction, followed by a demo/video of the results in different representative scenarios, followed by a careful performance analysis, and finally shortcomings of the approach.
It is OK to present this as a few slides with a demo/video for the results section, or to prepare a video for the entire presentation and then talk to the video as your presentation.
Please practice for timing; do not exceed five minutes.
As always, see How to make an attractive GitHub repo for tips on your README.md.
The final project is worth 50% of your final grade. The breakdown is:
- Milestone 1 & Progress: 20%
- Milestone 2 & Progress: 20%
- Milestone 3 & Progress: 20%
- Final submission: 20%
- All milestones presentations and final presentation: 20%
Feel free to share your successes, bloopers etc on social media. Patrick and I will retweet/share them. It'll be great to get the buzz going about your project. You may even get industry experts to chime in.
The main goal of final projects is to move the course forward and current with the industry. If you are trying to do something similar to year(s) old project, then you are starting behind.
These kind of projects, such as building on top of Pathtracer or doing new procedural generation, will be allowed if the additions are going to be novel and haven't been previously done. So the bar is set high.
Whereas, for projects like Vulkan, VR, DirectX Raytracing, the technology is recent and not much work has been done. So you will be the pioneers of these, so the barrier to entry is lower. Working on these will also open the possibility of the project becoming a regular project in the future.
Simulations, physically based animations will be allowed if you are implementing a novel technique that is vastly different from the seminal work in that field.
Be brave. There are a ton of online courses out there. Additionally, we'll try to connect you to any industry experts we know of.
It is completely ok to no do a graphics project. Not all useful software on GPU is grpahics related.
You can do a lot of things for final projects, including, but not limited to, some great graphs, comparisions with industry software etc.
The core piece of advice would be to take something really amazing you have done, and then port it to the GPU.
There are a ton of other topics you can think of as well. Machine Learning, deep learning, autonomous driving to name a few. Additionally, embedded systems with GPU such as Tegra or ARM GPUs, Apple Metal etc are also welcome.
It is all about doing projects using the GPU that are relevant to the industry and your field of interest.
For projects like machine learning etc, a source data is always required. There are usually a lot of open datasets available, most of which are a simple google search away.
Also try talking to other departments, professors or students who might have such data. There is always a good chance by approaching the right people.
If you still cannot find it, we can help connect you to some industry experts who might have such data.
Yes, as long as:
- There is no double-dipping (getting grades in 2 different courses for the same work).
- The goals are clearly layed out.
- The final project work is a GPU enhancement to other work.
- Other conditions that may be taken up case-by-case.
Missing milestones is NOT acceptable, except if you have matters such as interviews, recruitment etc, or obtain prior permission. In that case, do the following:
- Ensure your team presents on the day of the milestone presentations.
- Schedule a time with a TA (preferably Ottavio) and your team prior to the milestone presentations and discuss the progress made.