Rebuilding Sandcastle: Technical Implementation

[jjspace]

This issue will track the technical implementation of a new version of Sandcastle for CesiumJS. Design and features will primarily be worked out and discussed elsewhere. If you want to provide feedback on features you'd like to see please add them to our post on the community forum

High level plan

We plan to create a new workspace/package within our mono-repo to keep the new Sandcastle project separated but co-located with CesiumJS.

• itwin-ui: We expect to use the newest version of iTwin/Bentley UI library for both components and styling as much as possible. We may contribute improvements or components back to it as necessary.
• React: This is a given if we want to make use of iTwinUI.
• Monaco: We'll want to balance robust feature set with usability for users who prefer a more minimal code editing experience. This should greatly increase the DX of the editor bringing familiarity for any already used to using VSCode
• TypeScript: Unless there is a reason not to. This will be isolated to it's own package so it's separate from the rest of CesiumJS
• Build tools: We'll likely get started with the Vite plugin for react, and if needed we can use esbuild like the CesiumJS library itself

As we build out the new version we plan to have a beta/dev deployment for people to test against. More info on this as build systems and CI get set up

Running task list

I will try to keep this updated going forward as the "source of truth" for both large and small tasks that are remaining

• re-evaluate gallery search (PageFind?) New Gallery search for sandcastle #12755
• re-evaluate gallery system management
  • nested gallery structure implemented, mostly based on this comment
  • Migrate all existing sandcastles to the new structure (script to automate this added in Gallery conversion for Sandcastle v2 #12752)
• Code formatting command (added in Enable prettier formatter in monaco #12640)
• Integrate code generation "helpers" like "Add button" or "Add menu"
  • I think this will use snippets so it's there for power users but doesn't clutter the UI? need to investigate first iteration of snippets: Sandcastle helper snippets #12679
  • Feedback that they're really useful
  • Included in a dropdown in Sandcastle using Stratakit #12731, I think this works for now.
• Set up deployment to dev environment Sandcastle dev deployment #12680
• Finalize page title New sandcastle title and dirty state tracking #12794
  • Currently has a temporary port number or CI, probably should include the sandcastle title if set? Possibly should include a hint at the environment that's running
  • Use title of loaded gallery demo for now, drop CI to "hide" it
• Figure out how to display the version of CesiumJS and Sandcastle itself
  • Cesium version in top right seems nice so far, sandcastle version could be in the settings modal
• Warning for leaving the page with "unsaved" changes New sandcastle title and dirty state tracking #12794
• Finalize UI and implement new structure (design work tracked separately)
  • Resizable panels (is there an easy library in react for this? Yes) Add resizing panels to Sandcastle #12672
  • Feedback forum thread https://community.cesium.com/t/redesigning-sandcastle-requesting-feedback/41981/4
  • Sandcastle using Stratakit #12731
  • Clicking the title shows metadata information Sandcastle popovers #12832
  • Settings modal Add Sandcastle settings #12819
  • Clicking demo from gallery should load demo but not switch, add dedicated button to switch to code instantly
  • Add a link to the ref-docs on the page, maybe in the sidebar Sandcastle docs link #12834
  • Do a final pass on the app's CSS, all classes used, stratakit-vars where I can, logical structure and no "bleeding" styles Stratakit css cleanup #12802
  • Fix "Open code" button not working on an already loaded sandcastle Always switch to editor when it should #12820
• CSS only Stratakit for inside the bucket iframe Stratakit css cleanup #12802
• Add share details modal with the share URL to copy. Feature parity with what's in the current version Sandcastle popovers #12832
  • Improvements to sharing as a whole will be a follow up task
• Add "User settings" stored in local storage. Add Sandcastle settings #12819
  • Theme, light/dark/system (Sandcastle using Stratakit #12731)
  • Editor settings - A subset of all available for Monaco
    • Maaybe consider letting power users set the entire options object for monaco
    • Firacode font instead of Droid Sans and Font Ligatures has specifically been requested
• Default starting panel, gallery or editor? Add Sandcastle settings #12819
  • Default to gallery unless loading a specific example id or share key
  • Dedicated URL like index.html?new
  • can we make it selectible in the settings?
• Implement console output (started in Sandcastle using Stratakit #12731)
  • use for system messages too like missing types files or missing gallery files
  • Fix delayed output from hitting the logs after re-running comment
  • Fix multiple arguments Sandcastle console.log does not behave properly with multiple arguments #11141
  • Implement "stay scrolled" ourselves? (from comment)
  • verify performance of mirroring, I noticed it starting to lag a little when testing logs every frame
• Support loading URLs with a gist id in them for legacy urls that may still be out there Add support for loading GH Gist urls #12833
• Improve gallery search to include titles and descriptions. @ggetz was working on this after the initial search was implemented Sandcastle search and filtering updates #12854
• Monitor/update build system as needed (this is close but may still need a few adjustments)
  • Ensure we have a system for building against an arbitrary gallery location
  • Consider if vite is necessary or we'd be better off with esbuild directly
    • Verify how we're "building" cesium into the build, maybe we can point at a different path instead of copying built files.
  • Document and solidify build process from the repo root (related: comment)
  • finalize eslint config for new package files. First time TS and React is in there
• Content pass on all sandcastles
  • We may be able to cull a bunch of them. Others may need broken apart into multiple smaller ones
    • Specifically many of the ones under the development directory seem to be old versions or duplicates of others we have and don't need anymore
  • Re-evaluate labels and consider actual categories
  • Some work on this done in Sandcastle metadata cleanup #12865
  • Refine gallery "filter by label" presentation. We discussed having these be clickable pills so all labels are visible from the start
  • Remove Viewerless sandcastle if we deem it's not necessary anymore
  • Clean up styles for toolbars to make them consistent Consolidate toolbar styles for all sandcastles #12848
• Create structured plan to transition from current Sandcastle to the new one
  • The plan is to deploy the new to dev-sandcastle.cesium.com and point people to the new one, have both up for a bit with the old as default, Swap the default to the new with the old accessible for a period, then finally only provide the new one.
  • Determine if we want to push out intermittent releases for sandcastle only during the month of September, could be nice to respond to bugs quicker
    • Ideally not but will play it by ear
• Update CI to work in the cesium.com branch Sandcastle cross links #12855
• Add "Try new sandcastle" message to "old" sandcastle Sandcastle cross links #12855
• Add "Return to sandcastle" message to "new" sandcastle Sandcastle cross links #12855
  • Is there a way to link between the two using the share urls to preserve work?
    • No, not easily. Forward is fine, backward is not. see comment in Sandcastle cross links #12855
• Add feedback/bugs notice to new sandcastle before deployment Sandcastle cross links #12855

Post initial release:

• Follow up from gallery search improvements @ggetz Sandcastle search and filtering updates #12854
  • Part of this may be a discussion around the gallery structure and how it's imported into the app itself
• Add version info to Sandcastle for updates
  • Add "New features" type info panel for important releases? Maybe not necessary if it doesn't change much?
• Highlight editor lines for running sandcastle code and errors
• Expand Sandcastle API
  • Document the API itself, static page or overlay in Sandcastle?
  • Sliders are commonly needed
• Re-implement "new" detection for the gallery after the first release otherwise they will all show as new
  • Previously based on git diff, see comment
  • Maybe instead have a "addedIn" value in the sandcastle metadata instead of diff
• Insert snippets at cursor Stratakit css cleanup #12802 (comment)
• Prevent "Loading" message from jumping around Stratakit css cleanup #12802 (comment)
  • I've tried multiple things to mitigate this but I think it's still down to some sort of FOUC or resizing of the panels on page load or something weird. Might be worth moving the "Loading" message back to the top left so it doesn't rely on styles for positioning

PRs for this effort

• Sandcastle Reborn #12574
• Fix Sandcastle local build #12603
• Sandcastle helpers class #12619
• Add standalone page to Sandcastle v2 #12620
• Sandcastle - Nested gallery structure #12631
• Solidify new Sandcastle React structure #12639
• Enable prettier formatter in monaco #12640
• Add resizing panels to Sandcastle #12672
• Sandcastle helper snippets #12679
• Sandcastle dev deployment #12680
• Sandcastle using Stratakit #12731
• Gallery conversion for Sandcastle v2 #12752
• New Gallery search for sandcastle #12755
• Load monaco from packages not CDN #12760
• New sandcastle title and dirty state tracking #12794
• Stratakit css cleanup #12802
• Add Sandcastle settings #12819
• Always switch to editor when it should #12820
• Sandcastle popovers #12832
• Add support for loading GH Gist urls #12833
• Sandcastle docs link #12834
• Consolidate toolbar styles for all sandcastles #12848
• Sandcastle cross links #12855
• Sandcastle search and filtering updates #12854
• Sandcastle metadata cleanup #12865

Follow up tasks

A list of ideas we've had that could benefit Sandcastle but will not be considered part of the initial re-build effort

• An "embed" version of Sandcastle like CodePen or JSFiddle have that we can use in our documentation. Minimal editor + viewer
• Full Documentation integration
  • The better intellisense helps a lot but currently don't have a way to open the docs fully like before. Maybe we just recommend a second tab to people?
• Allow loading multiple CesiumJS versions
• Expand Gist/Github integrations - re: saving, sharing, commenting, collaborating
• Can we remove knockout dependency in all sandcastles? it may lead to users depending on it
• "Value monitor" type helper we can add to avoid spamming thousands of lines when wanting to track something that changes every frame for the console
• Implement console.debug mirroring (I think this is the only one missing that we'd want to implement other than group and trace)
• Expand share capabilities, what is saved, what is shared, how do things get shared
  • Is there an alternative "download" we can offer with the files that can be "dropped" back into sandcastle
  • Consider a "one click" option to switch between prod and local environment (maybe only if we detect the local server?)
• Save previous "successful" runs of a sandcastle to recover from crashes
  • Some discussion in this thread New sandcastle title and dirty state tracking #12794 (comment)
• Add a "Whats new" item to the sidebar to include update notes
• Consider option for "dropdown" or alternative from gallery card to "preview" code and provide more context

CC @ggetz

[jdfwarrior]

This will be a very welcomed update. Sandcastle has had the same look and feel for many years and could definitely use a refresh.

[ggetz]

Some running TODO's which have come up in discussion like the forum and @jjspace's initial PR.

[Moved to the top level post]

[jjspace]

Build structure

A big complication with this process is the actual build configuration. This is more complicated because locally we want to point at all the locally built files instead of copying them into the vite build statically. I wanted to share this breakdown I've been referencing a lot during my work on #12574. This is also not helped by the fact that when we deploy to prod we copy everything to the top level domain and basically all routes need to be adjusted somewhat to account for that. This will need to be kept in mind as we discuss how we want to change the import process into Sandcastles.

Local cesium server: npm start
    localhost:8080/Apps/Sandcastle/index.html
                                  /standalone.html (CESIUM_BASE_URL = '../../../Build/CesiumUnminified)
                                  /templates/bucket.html (CESIUM_BASE_URL = '../../../Build/CesiumUnminified') - look at load-es6.js
                                            /bucket.css (Accessed by '../templates/bucket.css')
                                  /gallery/Example.html (CESIUM_BASE_URL = '../../../Build/CesiumUnminified')
                                  /gallery/development/Example.html (still run in bucket.html or have custom `<base>` in standalone)

                                  /images/... (Accessed by '../images/...')
                  /Apps/SampleData/... (Accessed by '../../SampleData/...')

                  /Build/CesiumUnminified/Cesium.js ('../../Build/CesiumUnminified/index.js' from '../load-es6.js')
                                         /index.js
                                         /Assets/...
                                         /Widgets/...
                  /Source/Widgets/*.css (Accessed by '../../../Source/Widgets/...')

Production (check gulpfile buildSancastle for when isProduction)
    sandcastle.cesium.com/index.html
                         /standalone.html (CESIUM_BASE_URL = 'CesiumUnminified/')
                         /templates/bucket.html (CESIUM_BASE_URL = '../CesiumUnminified/')
                                   /bucket.css (Accessed by '../templates/bucket.css')
                         /gallery/Example.html (CESIUM_BASE_URL = '../CesiumUnminified/')

                         /images/... (Accessed by '../images/...)
                         /SampleData/... (Accessed by '../SampleData/...')

                         /CesiumUnminified/Cesium.js (Accessed by '../CesiumUnminified/...')
                                          /Assets/...
                                          /Widgets/...

CI deployment
    ci-build.cesium.com/cesium/\[branch\]/Apps/Sandcastle/index.html
                                                         /templates/bucket.html (CESIUM_BASE_URL = '../../../Build/CesiumUnminified/')
                                                                   /bucket.css
                                                         /gallery/Example.html (CESIUM_BASE_URL = '../../../Build/CesiumUnminified/')

                                                         /images/... (Accessed by '../images/...')
                                         /Apps/SampleData/... (Accessed by '../../SampleData/...')
                                         /Build/CesiumUnminified/index.js
                                                                /Assets/...
                                                                /Widgets/...

Local vite dev: npm run dev
    localhost:5173/index.html
                  /templates/bucket.html (CESIUM_BASE_URL = '/Build/CesiumUnminified')
                            /bucket.css
                  /images/... (Accessed by '../images/...')
                  /SampleData/... (Accessed by '../../SampleData/...')
                  /Build/CesiumUnminified/Cesium.js
                                         /Assets/...
                                         /Widgets/...

Local vite build for server: npm run build-app -> Apps/Sandcastle2 -> npm start
    localhost:8080/Apps/Sandcastle2/index.html
                                   /templates/bucket.html (CESIUM_BASE_URL = '/Build/CesiumUnminified')
                                             /bucket.css
                                   /images/... (Accessed by '../images/...')
                  /Apps/SampleData/... (Accessed by '../../SampleData/...')
                  /Build/CesiumUnminified/Cesium.js (Accessed by '/Build/CesiumUnminified')
                                         /Assets/...
                                         /Widgets/...
                  /Source/Cesium.d.ts

There are basically 3 main ways a specific sandcastle can be loaded. They all need to work. More specifically routes to imported JS or CSS and sample data need to work for all of these routes regardless of the environment outlined above.

• Directly by navigating to the gallery file sandcastle.cesium.com/gallery/3D Models.html
• bucket.html which is the iframe used in the main sandcastle UI sandcastle.cesium.com/templates/bucket.html
  • bucket-requirejs.html which is the iframe location that uses load-cesium-es6.js to load the esm module instead of the iife. This is only used locally and gets replaced when deployed
  • Note that bucket.html is "nested" inside the templates directory which lets paths from it mimic those in the gallery directory, this is important for them to both work the same
• standalone.html which is the full standalone page that is capable of loading a sandcastle from the URL string https://sandcastle.cesium.com/standalone.html#c=bY...

Gallery files

Another big topic we still need to settle on is the structure of an actual, saved, gallery file. Right now they're all html files that work on their own. The code is extracted from them and loaded into the code editors when they're loaded in Sandcastle. Metadata like the title, tags and description is included in the <head> tags.

Open questions:

1. Should these remain as singular html files? or should they be broken apart into html + js files?
2. Should every sandcastle example have a @import(bucket.css) statement or can this be moved to the bucket.html and gallery files themselves?
3. Is there a better way we can extract information to avoid loading every single gallery item on every page load?
4. How do we manage more metadata?
   • There's been discussion of at the very least longer descriptions
   • I also think more integrated "explainers" for lines or chunks of code could be nice but I don't know the best way to integrate those
5. Is there anything more we should include in the wrapper for sandcastle examples that could help the application as a whole work better?
   • For example forcing the iframe window to have a cesiumViewer variable accessible so the surrounding app can access it

CC @ggetz there's more beyond these to talk about but hopefully this can help kickstart our discussions

[ggetz]

@jjspace Re: Gallery files

Before we get into the technical aspects of the gallery file (like the JS/CSS/HTML breakdown, import styles, etc), I think it would be helpful to decouple what we'd like out of a gallery example and how to organize them from implementation itself.

To start what makes up a gallery example? I think defining this will help guide (3) and is directly involved with (4).

• Currently, sandcastle examples consist of the following, correct?
  • The code
  • A thumbnail
  • A title
  • A description
  • Labels (or tags)
• In addition to tags, of which there can be multiple, would it be helpful to organize the examples into categories (or even subcategories if appropriate)? That could help with organization and user discoverability because each category can be more intentional or curated than the current deluge of examples in each tag.
• I do like the idea of having more digestible "explainer" content to augment a sandcastle. IMO this should be separate from the description as that should be more concise. The explainer could potentially be a text property. But maybe it could be something with a bit more expressiveness such as markdown.
• Are there any other metadata fields which we would like to maintain?

---

Then going one level deeper, how do we ideally want to store that metadata (4)? I'm putting aside backwards compatibility for a moment to think about how it might make sense if starting from scratch.

• I think it's a bit cumbersome to manage this all in an HTML file directly. Contributors often forget to update descriptions and tags because they're easy to miss.
• Ideally, a more intentional approach to the metadata could help us avoid loading each entire sandcastle example on page load (3).
• Maybe we can use yaml for capturing gallery example metadata. YAML is common in a lot of documentation tools and static site generators, so I don't think it should be a barrier to entry for contributors. For instance:

title: Adjust 3D Tileset Height
thumbnail: "./3d-tiles-adjust-height.jpg"
description: Adjust the relative height of a tileset to better align it with terrain.
labels: 
   - 3D Tiles
   - Tutorial
category: Working with 3D Tiles
context: Each instance of a 3D tileset has a modelMatrix property that, when set, modifies the entire tileset's position, rotation, and scale. This transformations are applied in worldspace before the tileset's root tile transform.

Or, if we want more formatting, that yaml can be ported to a markdown file's front matter:

---
title: Adjust 3D Tileset Height
thumbnail: "./3d-tiles-adjust-height.jpg"
description: Adjust the relative height of a tileset to better align it with terrain.
labels: 
   - 3D Tiles
   - Tutorial
category: Working with 3D Tiles
---

Each instance of a 3D tileset has a `modelMatrix` property that, when set, modifies the entire tileset's position, rotation, and scale. This transformations are applied in worldspace before the tileset's [root tile transform](https://github.com/CesiumGS/3d-tiles/tree/main/specification#transforms).

---

For (5), I'm having trouble picturing a use case, but it sounds like you have some in mind. Could you share what you were thinking about?

Let me know your thoughts on all that, and then we can talk through the details of how we'd like to store the code and what this may mean for backwards compatability.

[jjspace]

@ggetz sorry my response got a little lost but here's my thoughts

To start what makes up a gallery example?

Correct, that is the current list, even if not all of those are displayed promenantly (description).

I think it could be helpful to include a general category but I'm not sure how different it would be than tags. It would definitely help to categorize in whatever UI we pick but I could also see it being an issue that certain sandcastles should belong to multiple categories and then we're right back to tags.
Maybe we just pick a set of tags that are "important" that we can use to categorize in the UI but still only store them as tags?

how do we ideally want to store that metadata?

I think it's a bit cumbersome to manage this all in an HTML file directly. Contributors often forget to update descriptions and tags because they're easy to miss.

I agree it may get cumbersome to all be in the same html file. I also think the more we do in there the harder it may make our lives to try and parse/process it when we want to load examples. That said, a separate file may end up being even easier to miss and forget to update but I think that's going to be true for any solution we come up with.

Ideally, a more intentional approach to the metadata could help us avoid loading each entire sandcastle example on page load (3).

I agree, even if we stuck to a single file I would want to pre-process it more so we can load only the metadata on page load instead of the full example.

Maybe we can use yaml for capturing gallery example metadata.

I do really like the idea of using YAML, it's common and easy to use and can be pretty extensible. Definitely easier to write and read than JSON. The main question will be where does it live.

This is all pointing to having one or more "sidecar files" for a given sandcastle.
We have also discussed adding a larger "explainer" type block of text or MD so that needs considered as well. Maybe that could be where we use Frontmatter for the metadata and the rest of the file for any explainer if it exists? If a sandcastle doesn't need that maybe it's just an adjacent [sandcastle name].yaml that has the data?

One big question I think we need to answer is still: Should people be able to load gallery examples directly? If the answer is "yes" then I think we should keep what we can inside the html file itself. If the answer is "no" then we can do more with the "save format". Maybe even separating into dedicated JS and HTML files. or creating subdirectories for each sandcastle and use the file structure itself to encapsulate the metadata sidecar files and the thumbnail and the code etc.

For (5), I'm having trouble picturing a use case, but it sounds like you have some in mind. Could you share what you were thinking about?

The primary usecase I like is just having access to the active Viewer from the console in the outer window context. It's also really useful if we want to add any sort of helper buttons to the Sandcastle UI itself to add code like "Save the current view" which I've found very helpful. Without a consistent way to reach into the iframe or for the iframe to reach out it's impossible to do things like that.
Interestingly I found that Babylon's playground requires that the code creates a createScene function in order to run. Maybe we should do the same with viewer?

[ggetz]

Categories

Maybe we just pick a set of tags that are "important" that we can use to categorize in the UI but still only store them as tags?

I agree this is valid from an implementation standpoint. From a usability perspective, just displaying the tags as "categories" is a bit overwhelming due to the number of them, and I think we would like to be more intentional about which ones we present and in which order.

I agree it may get cumbersome to all be in the same html file. I also think the more we do in there the harder it may make our lives to try and parse/process it when we want to load examples. That said, a separate file may end up being even easier to miss and forget to update but I think that's going to be true for any solution we come up with.

One big question I think we need to answer is still: Should people be able to load gallery examples directly?

Good point. Just for clarity here, you're talking about the ability to take the HTML file and load it outside of the context of Sandcastle as more of a standalone example, correct? Or are you looking more to address "saving" a sandcastle to a file and being able to bring it back in?

Overall, I'm personally favoring a gallery example being a directory of files for the flexibility and simplicity from the point of view of the user. Theoretically, this wouldn't prevent the example from working as a standalone example. And we could save as a zip file if the user wants to download, while an example could be brought back in by unzipping in browser (for example).

• [Optional] README.md for the explainer text so we can use markup, link to other resources, etc. This also gives us the benefit of having the explainer info be discoverable and browsable in the GitHub repo as well as Sandcastle.
• [Optional] Thumbnail image
• HTML
• JS
• [Optional] Standalone CSS
• yml (or frontmatter in the README.md) specifying all the needed metadata:
  • title (string),
  • description (string),
  • tags (string[]),
  • html (string, relative path),
  • js (string, relative path),
  • css (string, relative path, optional)
  • thumbnail (string, relative path, optional),

Potentially, we could include things like version info, author, last updated, or search embeddings if they prove useful.

I really am leaning towards the yml data, as it will be easy to generate a gallery list with all the needed info for displaying the example in the Sandcastle UI without having to load to content itself.