iris-hep · alexander-held · Nov 29, 2024 · Nov 29, 2024 · Nov 29, 2024 · Nov 29, 2024
diff --git a/_data/people/alexander-held.yml b/_data/people/alexander-held.yml
@@ -1,5 +1,6 @@
 active: true
 e-mail: [email protected]
+challenge-area: analysis-grand-challenge
 focus-area:
   - as
 inspire-id: Alexander.Held.1

diff --git a/_data/people/oshadura.yml b/_data/people/oshadura.yml
@@ -1,4 +1,5 @@
 active: true
+challenge-area: analysis-grand-challenge
 focus-area:
   - doma
   - as
@@ -8,6 +9,7 @@ photo: "/assets/images/team/Oksana-Shadura_.jpg"
 shortname: oshadura
 title: Analysis Grand Challenge Co-coordinator
 website: https://github.com/oshadura
+
 presentations:
   - title: Tuning the CMS Coffea-casa facility for 200 Gbps Challenge
     date: 2024-10-24

diff --git a/_includes/navbar.html b/_includes/navbar.html
@@ -41,6 +41,10 @@
                         {% endfor %}
                         <div class="dropdown-divider"></div>
                         <a class="dropdown-item" href="/grand-challenges.html">Grand Challenges</a>
+                        {% assign activities = site.pages | where: "layout", "challenge-area" | sort_natural: 'title' %}
+                        {% for mypage in activities %}
+                        <a class="dropdown-item" href="{{mypage.permalink}}">{{mypage.title}}</a>
+                        {% endfor %}
                         <div class="dropdown-divider"></div>
                         <a class="dropdown-item" href="/impact-beyond-hep.html">Impact Beyond HEP</a>
                         <a class="dropdown-item" href="/outreach.html">Education and Outreach</a>

diff --git a/_layouts/challenge-area.html b/_layouts/challenge-area.html
@@ -0,0 +1,165 @@
+<!doctype html>
+
+{% include layout_header_navbar.html %}
+
+    <div id="container">
+      <div class="container page">
+        <main role="main">
+
+{% assign page_challenge_area = page.name | basename %}
+{% assign sorted = site.pages | where: "pagetype", "project" | where_exp: "item", "item.challenge-area contains page_challenge_area" | smart_title_sort %}
+{% assign fa_team = site.data.people | where_exp: "item", "item.challenge-area contains page_challenge_area and item.active and item.hidden != true" | last_name_sort: "name" %}
+{% assign fa_fellows = site.pages | sort_natural: 'title' | where_exp: "item", "item.pagetype == 'fellow' and item.challenge-area contains page_challenge_area and item.hidden != true" %}
+{% assign pres_list = site.data['sorted_presentations'] | where_exp: "item", "item.challenge-area contains page_challenge_area"  %}
+{% assign pubs = site.data['sorted_publications'] | where_exp: "item", "item.challenge-area contains page_challenge_area" %}
+{% include get_selected_videos.html nvideos=6 challenge_areas=page_challenge_area %}
+
+{% assign quicklinks = "" | split: "," %}
+{% if fa_fellows.size > 0 %}
+{% assign quicklinks = quicklinks | push: '<a href="#fellows"> Fellows</a>' %}
+{% endif %}
+{% if sorted.size > 0 %}
+{% assign quicklinks = quicklinks | push: '<a href="#projects"> Projects</a>' %}
+{% endif %}
+{% if pres_list.size > 0 %}
+{% assign quicklinks = quicklinks | push: '<a href="#presentations"> Presentations</a>' %}
+{% endif %}
+{% if pubs.size > 0 %}
+{% assign quicklinks = quicklinks | push: '<a href="#publications"> Publications</a>' %}
+{% endif %}
+{% if selected_videos.size > 0 %}
+{% assign quicklinks = quicklinks | push: '<a href="#videos"> Videos</a>' %}
+{% endif %}
+{% if quicklinks.size > 0 %}
+  Jump to: {{ quicklinks | join: " - " }}
+{% endif %}
+
+<div>
+<br>
+<center><h1>{{ page.title }}</h1></center>
+
+{{content}}
+</div>
+
+
+{% if fa_team.size > 0 %}
+<div class="container-fluid">
+  <div class="row">
+    {% for person in fa_team %}
+    <div class="card" style="font-size:10px;width: 87px;height: 135px;margin-bottom:-15px; border: none; text-align: center;">
+      <img style="width: 85px;height: 85px; object-fit:cover;" src="{{person.photo}}"  alt="Card image cap">
+      {%- if person.website -%}
+        <a href="{{person.website}}">{{ person.name }}</a>
+      {%- else -%}
+        {{ person.name }}
+      {%- endif -%}
+      </div>
+    {% endfor %}
+  </div>
+  <br>
+</div>
+{% endif %}
+
+{% if fa_fellows.size > 0 %}
+<h2 id="fellows" class="alt-h2 text-center mb-3 mt-lg-6" id="pres">Current and Previous {{page.short_title | upcase }} Fellows</h2>
+
+<div class="container-fluid">
+  <div class="row">
+    {% for person in fa_fellows %}
+    <div class="card" style="font-size:10px;width: 87px;height: 135px;margin-bottom:-15px;">
+      <img style="width: 85px;height: 85px; object-fit:cover;" src="{{person.photo}}"  alt="Card image cap">
+        <a href="{{person.permalink}}">{{ person.fellow-name }}</a>
+      </div>
+    {% endfor %}
+  </div>
+  <br>
+</div>
+{% endif %}
+
+
+{% if sorted.size > 0 %}
+<div class="container-fluid projects">
+  <h2 id="projects"  class="alt-h2 text-center mb-3 mt-lg-6" id="projects">{{ page.short_title | upcase }} Projects</h2><br>
+<center>
+  <div class="row">
+{% for mypage in sorted %}
+<div class="card" style="width: 17rem;">
+<a href=/projects/{{mypage.shortname}}.html>
+<img class="card-img-top" src="/assets/{{mypage.image | default: "logos/Iris-hep-5-just-graphic.png"}}" alt="Card image cap">
+</a>
+<div class="card-body d-flex flex-column">
+<div class="card-text">
+<b><p><font size="+1">{{mypage.title}}</font></p></b>
+{{mypage.blurb}}<br>
+<a href=/projects/{{mypage.shortname}}.html> More information</a><br>
+{%- include maturity.html project=mypage -%}
+</div>
+</div>
+</div>
+{% endfor %}
+  </div>
+</center>
+  <br>
+</div>
+{% endif %}
+
+{% if pres_list.size > 0 %}
+<section class="">
+  <div class="container-lg p-responsive py-5 py-md-6 ">
+    <h2 id="presentations" class="alt-h2 text-center mb-3 mt-lg-6" id="pres">{{page.short_title | upcase }} Presentations</h2>
+    {% expandable pres_list 15 %}
+      {% include print_pres.html talk=expandable %}
+    {% endexpandable %}
+</div>
+</section>
+{% endif %}
+
+{% if pubs.size > 0 %}
+<section class="">
+<div class="container-lg p-responsive py-5 py-md-6 ">
+    <h2 id="publications" style="margin-top: -110px;" class="alt-h2 text-center mb-3 mt-lg-6" id="pubs">{{ page.short_title | upcase }} Publications</h2>
+    {% expandable pubs 15 %}
+      {% include print_pub.html pub=expandable %}
+    {% endexpandable %}
+</div>
+</section>
+{% endif %}
+
+{% if selected_videos.size > 0 %}
+<h2 id="videos" class="alt-h2 text-center mb-3 mt-lg-6" id="pubs">Recent {{ page.short_title | upcase }} Recordings</h2>
+{% include display_videos.html videos=selected_videos %}
+{% endif %}
+
+<section class="">
+  <div class="container-lg p-responsive py-5 py-md-6 ">
+    <h2 class="alt-h2 text-center mb-3 mt-lg-6" id="join-us">Join us</h2>
+    <p class="alt-lead text-gray text-center col-md-10 mx-auto">We collaborate with groups around the world on code, data, and more. See our project pages for more.</p>
+  </div>
+  <div style="overflow: hidden;">
+    <div class="featured-orgs text-center p-2">
+      {% for org_list in page.collaborators   %}
+        {% assign org_hash = org_list | first  %}
+        {% assign org = org_hash[1] %}
+        <a href="{{org.link}}"><img src="/assets/images/{{org.image}}" height="100" alt="{{org_hash[0]}}"> </a>
+      {% endfor %}
+    </div>
+  </div>
+</section>
+
+{% if sorted.size > 0 %}
+  {{ sorted | gantt_project: page.title }}
+
+  <script>
+    mermaid.initialize({ startOnLoad: true });
+  </script>
+{% endif %}
+
+
+        </main>
+      </div>
+      {% include_cached footer.html %}
+    </div>
+
+  {% include_cached layout_analytics.html %}
+  </body>
+</html>
diff --git a/_scripts/challenge_area.rb b/_scripts/challenge_area.rb
@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+require 'yaml'
+require 'pathname'
+require 'set'
+
+def ensure_array(arr)
+  return [] if arr.nil?
+
+  arr.is_a?(Array) ? arr : [arr]
+end
+
+project_files = Pathname.glob('pages/projects/*.md')
+counter = {}
+project_files.each do |fn|
+  proj = YAML.load_file fn
+
+  fa = ensure_array proj['challenge-area']
+  team = ensure_array proj['team']
+  team.reject! { _1.include?(' ') }&.reject! { _1.include?('@') }
+
+  fa.each { |a| team.each { |t| counter[t] = (counter[t] || Set.new) << a } }
+end
+
+people_files = Pathname.glob('_data/people/*.yml')
+people_files.each do |fn|
+  info = YAML.load_file fn
+
+  user_fa = info['presentations']&.flat_map { ensure_array(_1['challenge-area']) }&.to_set&.delete('core')
+  proj_fa = counter[info['shortname']]&.delete('core')
+  puts "#{info['name']}: #{user_fa.to_a} | #{proj_fa.to_a}"
+  all_fa = (user_fa || Set.new) | (proj_fa || Set.new)
+  info['challenge-area'] = all_fa.to_a unless all_fa.empty?
+
+  sorted = info.sort_by { _1 }.to_h
+  sorted['presentations'] = sorted.delete 'presentations'
+
+  fn.write(sorted.to_yaml.lines[1..].join)
+end
diff --git a/_scripts/people.schema.json b/_scripts/people.schema.json
@@ -11,6 +11,7 @@
       "type": "array",
       "items": { "$ref": "#/definitions/focus_area" }
     },
+    "challenge-area": { "$ref": "#/definitions/challenge_area" },
     "inspire-id": { "type": "string" },
     "institution": { "type": "string" },
     "name": { "type": "string" },
@@ -44,6 +45,9 @@
     "focus_area": {
       "enum": ["as", "ia", "ssc", "doma", "ssl", "osglhc", "core"]
     },
+    "challenge_area": {
+      "enum": ["analysis-grand-challenge"]
+    },
     "focus_areas": {
       "oneOf": [
         { "type": "null" },
@@ -62,6 +66,7 @@
         "meeting": { "$ref": "#/definitions/empty_string" },
         "meetingurl": { "$ref": "#/definitions/empty_string" },
         "focus-area": { "$ref": "#/definitions/focus_areas" },
+        "challenge-area": { "$ref": "#/definitions/challenge_area" },
         "project": { "$ref": "#/definitions/string_or_array" },
         "location": { "$ref": "#/definitions/empty_string" },
         "comment": { "$ref": "#/definitions/empty_string" },

diff --git a/pages/challenge-areas/analysis-grand-challenge.md b/pages/challenge-areas/analysis-grand-challenge.md
@@ -0,0 +1,44 @@
+---
+permalink: /analysis-grand-challenge.html
+layout: challenge-area
+short_title: agc
+title: Analysis Grand Challenge
+pagetype: challenge-area
+---
+
+## Analysis Grand Challenge
+
+The most up-to-date documentation about the Analysis Grand Challenge is located at this website: [https://agc.readthedocs.io/en/latest/](https://agc.readthedocs.io/en/latest/?badge=latest)
+
+The large increase in data volume at the HL-LHC requires rethinking how physicists interact with the data when developing and performing analysis. In addition to raw throughput, it is critical that analysis systems are flexible, easy to use, and have low latency to facilitate the design stages. The Analysis Grand Challenge was designed to span the scope of Analysis Systems focus area, transverse a vertical slice through the tools being developed by the Analysis Systems focus area, and increase intra-Institute connections with DOMA and SSL. The goal is to demonstrate that the analysis system can not only cope with the increased data volume, but can also deliver enhanced functionality compared to the analysis systems used at the LHC today. The challenge is formulated as a user story with assumptions, and acceptance criteria.
+
+The Analysis Grand Challenge includes both integration of software components for analyzing the data as well as the deployment of the analysis software at analysis facilities. The vertical slice implements the functionality needed for a prototypical analysis use case with a moderately complex analysis with multiple event selection requirements, observables to be histogrammed, and systematic uncertainties that must be taken into account. The image below gives an overview of the software tools that must be integrated for this vertical slice.
+
+{% include figure.html
+  alt="Analysis Systems Scope"
+  file="assets/images/cabinetry-vertical-slice.png"
+  class="center"
+  figure-style="width:100%"
+%}
+
+
+In addition, the challenge incorporates enhanced functionality relative to the  the analysis systems used at the LHC today
+- End-to-end analysis optimization including systematics on a realistically sized HL-LHC (∼200 TB) end-user analysis dataset
+- Analysis Preservation & Reinterpretation: The ability to preserve the optimized analysis (in git repositories, docker images, workflow components, etc.), reproduce results, and reinterpret the analysis with a new signal hypothesis.
+
+
+The inclusion of differentiable programming, a relatively new concept in HEP, into the challenge carries some risks. We note however that it has the potential to move the field forward in several important ways:
+
+- Intellectual Leadership: It is a modern paradigm growing and abstracting from the success of deep learning, and a more natural fit to HEP than replacing everything with machine learning.
+- Increased Functionality: We will have more sensitive analyses. Differentiable analysis systems
+would accelerate and improve essentially all fitting / tuning / optimization tasks. It also facilitates propagation of uncertainty in a more powerful way and paves the way to hybrid systems that fuse traditional approaches and machine learning more seamlessly.
+- Connection with Industry: This has been an effective conduit to connections with Google (Jax and Tensorflow teams) and the pytorch community.
+- Foster Innovation: Differentiable programming opens up a new range of possibilities for performing analysis in physics at the HL-LHC.
+- Training & Workforce development: Young people entering the job market with Machine Learning and Differentiable Programming skills will have a unique and valuable skill set. Differentiable programming will force physicists to take an end-to-end approach to problem solving, a skill that is already looked for both within and outside HEP.
+
+This challenge involves milestones and deliverables in DOMA, Analysis Systems, and SSL. Year 3 will include a Blueprint and other meetings focused on scoping of the target analysis, the needed capabilities of an analysis system, and roadmaps for how the components will interact. Year 4 will include initial benchmarking of analysis system components and integrations, and we aim to execute the analysis grand challenge in Year 5.
+
+
+The video below provides an overview of the tools being developed by the Analysis Systems focus area, the deployment of those tools on analysis facilities, and the integration of these efforts in the context of the Analysis Grand Challenge.
+
+<iframe width="560" height="315" src="https://www.youtube.com/embed/nDpk0tYOVlc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
diff --git a/pages/challenge-areas/data-grand-challenge.md b/pages/challenge-areas/data-grand-challenge.md
@@ -0,0 +1,22 @@
+---
+permalink: /data-grand-challenge.html
+layout: challenge-area
+title: Data Grand Challenge
+---
+
+# Data Processing Grand Challenge
+
+During a nominal year of HL-LHC data taking, ATLAS and CMS together expect to take close to one exabyte of RAW data. Both experiments intend to process each year’s worth of data as early as possible in the year after. A reasonable working assumption is thus that one exabyte of data across both experiments will have to be processed in 100 days, or roughly 10 PB/day, or 1 Tbit/sec.
+
+The RAW data will reside on tape archives across the Tier-1s and CERN, and will get processed at Tier-1s, Tier-2s, and HPC centers. It is highly likely that the two experiments will overlap in time and at least some processing locations, e.g. the large DOE and NSF HPC centers. And it is virtually guaranteed that both will overlap on many network segments worldwide.
+
+IRIS-HEP, together with the US LHC Operations programs, the ATLAS and CMS global collaborations, and the WLCG
+arrived at a series of data challenges for the next several years (2021, 2023, 2025, 2027),
+during which the capabilities and performance of the global infrastructure will be slowly scaled out to reach HL-LHC requirements.
+This includes three levels of challenges that interleave and build on each other.
+First, there will be functionality evaluations during which new functionality of various infrastructure software products are tested.
+Second are scalability challenges of such individual products,
+and third, global production challenges in alternate years during which the production systems are exercised at increasing scale.
+The first two types of challenges feed into the third as new products providing new functionality, or scale,
+enter the production systems over time.
+IRIS-HEP is engaged in these challenges at all levels via projects in multiple of its focus areas.
diff --git a/pages/challenge-areas/training-grand-challenge.md b/pages/challenge-areas/training-grand-challenge.md
@@ -0,0 +1,15 @@
+---
+permalink: /training-grand-challenge.html
+layout: challenge-area
+title: Training Grand Challenge
+---
+
+# Training Grand Challenge
+
+We are now working to define, with the larger community, a series of specific goals for the period 2021-2023 in four categories and to work with the community to achieve them.
+
+  * **Scalability** - We aim for sufficient scalability in the training activities such that all students and postdocs can receive training in both the introductory material and the more advanced material. In the steady state we expect a required scale approximately equal to the number of incoming students each year.
+  * **Sustainability** - We aim to develop community processes by which both the instructors involved in training activities, and the training materials themselves, are continually renewed and meet the other two goals.
+  * **Training Scope** - We aim for a curriculum (introductory, intermediate, advanced) that broadly meets the needs of the community and evolves over time as needed.
+  * **Diversity and Inclusion** - The participation in the training should be representative of our community and (as we engage earlier in the pipeline) should work to represent the society at large
+