(click to enlarge slide)

Five years ago, soon after I was sworn in as the Director of the Office of Science, it was my great pleasure to speak to SC02 about the Department’s plans for ultra-scale computing.  Today, I have the pleasure of reporting what we have accomplished so far and about the exciting road ahead.

At my age, five years isn’t such a very long time and yet so much has changed that it is hard to imagine. 

Five years ago, the Office of Science, and the physical sciences in general, had seen nearly a decade of stagnation and flat budgets.

Five years ago, the largest computer available to the Office of Science was the NERSC facility, which seemed quite capable at 3 Teraflops peak, and our highest sustained speed on, for example, fusion codes was 485 Gigaflops.  Data was traveling into NERSC, and around the DOE system, via ESnet at 622 megabits per second.  It is hard to imagine that this was only five years ago and that NERSC had transitioned from vector machines to massively parallel machines less than five years before that.

Five years ago, we were also awestruck by the great accomplishment of the Japanese with their Earth Simulator in the Spring of 2002.  I give the Japanese full credit for recognizing the power of scientific simulation because the Earth Simulator really opened people's eyes.  At least, it opened my eyes – to the possibility of doing things on a scale that we had never done before and more rapidly than we had imagined.

On June 22 of that year, I gave the commencement address at the Rand Graduate School.  In that talk, I stated: “In science of the 21st century, simulation and high-end computation are equal partners with theory and experiment.  Scientific leadership, the basis for our economic, physical, and intellectual prosperity depends on this triad, our being first in each component.”

(click to enlarge slide)

So at SC02, I announced the Department of Energy’s vision for scientific computing through two new initiatives: Scientific Discovery through Advanced Computing, or SciDAC the year before, and UltraScale Computation – which we now call Leadership Computing Facilities – to be launched in 2003.  I also hinted at my vision for true scientific computing user facilities that would be open to all, including industry, and prioritize access based on peer review – similar to the light sources and particle accelerators the DOE had provided for decades.

As you well know, in computing a lot can change in five years. But I can tell you that we did not waiver from our vision and our commitment to realizing high performance computing as a true third pillar for scientific discovery.  So before I tell you where our vision is now, I want to tell you what we have accomplished thus far.

This month, NERSC has accepted a 100TF Cray XT-4 system. With over 2,500 users, NERSC was in 2002, and is today, the best run large capacity computing facility in the world with a remarkable record of reliability and user support. ESnet delivers 10 gigabit per second core service and Metropolitan Area Networks, such as the one in the Bay Area that moves data into and out of NERSC, at 20 gigabits per second.

NERSC and ESnet have proved so reliable that OMB told us to drop their reliability metric since we were always reporting five 9’s (99.999%) of reliability and the government wide goal for scientific user facilities is 90%.  So although I still see our scientific computing user facilities as true scientific user facilities, obviously they are also unique.

NERSC continues to play a critical role in filling the ever growing demand within the Office of Science for high performance computing.  But they also provide assistance in scaling applications to take full advantage of the resources available at NERSC and helping users to “outgrow” NERSC.

This is possible because the Department now also offers two Leadership Computing Facilities – at Oak Ridge National Laboratory and Argonne National Laboratory.  Together these facilities offer architectural diversity, with Cray systems at Oak Ridge and IBM Blue Gene systems at Argonne.  Today, the Argonne Blue Gene is delivering 5.7TF peak with sustained performance on materials simulations up to 60%, and the Oak Ridge Cray XT system is delivering 119TF peak with sustained performance on fusion codes at up to 75%.

This is really exciting because back in 2002, I talked about fusion simulations and how important simulation would be to realizing fusion for clean energy.  Back then, the state of the art was barely 2-dimensional, and we thought we would need hundreds of sustained teraflops to go to 3-D.  But today, at the Oak Ridge Leadership Computing Facility, the AORSA code runs a fully 3-dimensional simulation of confined plasma that is being validated with experimental observations.  The code is demonstrating radial wave propagation with rapid absorption and efficient plasma heating as we saw in experiments.  But new insights are also being gleaned.  We are seeing “hot spots” near the antenna surface and “parasitic draining” of heat to the plasma surface in smaller reactors. These simulations are today shedding new light on the behavior of superheated ionic gas which will help us understand and control the multibillion-dollar ITER fusion reactor.

The AORSA code is using 22,500 processors.  This team has acknowledged considerable help from SciDAC and the Oak Ridge Leadership Computing Facility to get to where they are today, which is so much further then we could have imagined in 2002.  In fact, we started SciDAC in 2001 because we realized that the power of massively parallel processors brought with it challenges in use and efficiency.  Sustained speeds on some problems are as much as 75% of peak, while on other important scientific problems the efficiencies are less than 10%. Back in 2002, the lion share of applications fell into the latter category, and we knew we had to do something to change that if scientific computing were to really become a third pillar that all scientists could use.

SciDAC has delivered some remarkable results, improving code performance by up to 10,000% and taking simulations from 1 and 2 dimensions to full 3-D in areas such as astrophysics, fusion, and combustion.  In 2006, we expanded SciDAC to other areas important to the Department, such as modeling the reactive flow of contaminants in groundwater and computational biology focused on accelerating bio-energy options such as cellulose-based ethanol.  SciDAC has really made a difference in effectively utilizing our new leadership machines but also contributes to the success of our other investments.

As we look to the future, the Office of Science will soon launch a fusion simulation project that will integrate these efforts and deliver an unprecedented new tool for fusion research.  This is similar to what we are seeing today in climate research.

Five years ago in climate simulation, the best scale available to climate scientists in the United States was a computational grid 100 km X 100 km.  Over those lengths, mountains, hurricanes, and coastlines were averaged out.  Further, the codes were very fragmented and separate – there wasn’t even a single unified ocean model, and none of the ocean models were integrated with the cloud models or the upper atmosphere models, and we had barely started thinking about the carbon cycle.

The first fully-coupled integrations of atmosphere with ocean models at 70km scale were begun last year at Oak Ridge.  This is a significant increase in resolution from the 140km resolution used in the last Intergovernmental Panel on Climate Change (IPCC) simulation.  The smaller scale should do a much better job of resolving tropical cyclones, an important mechanism for heat and water transport in the climate system, which were poorly simulated in the IPCC simulations.

Three 100 year fully coupled 1990 Control Runs of the Community Climate System Model, or CCSM, were also carried out at Oak Ridge in just under three days, using different values of snow albedos in the sea ice component, to determine which values gave the best simulation of sea ice in the Arctic Ocean. All three runs demonstrated marked improvements in the El Nino/Southern Oscillation simulations.  This has greatly reduced some of the outstanding model biases in the CCSM and moved us toward some important issues such as regional surface hydrology of land which is critical for drought predictions.  They are now in position to test the full carbon-nitrogen cycle.  This is remarkable progress in climate modeling and well beyond our dreams of five years ago.

The climate community is rapidly moving toward a fully coupled global climate model with dynamic ecosystem feedback and predictive capability at regional scales.  They are not there yet. But this is now within reason to reach for because of the unbelievable progress we have achieved over the past five years.

Which leads to where we are going.

(click to enlarge slide)

After a decade of flat budgets, the Office of Science is expected to grow as part of the American Competitiveness Initiative.  High Performance Computing, advanced networks and the core research in math and computer science that supports advance are a key element in this initiative.

Argonne recently announced the purchase of a 500TF IMB Blue Gene P system to be available in 2008, and Oak Ridge is scheduled to receive a 1 Petaflop Cray Baker System at this time next year.  To ensure that science is ready to make good use of these machines on day one, we have supplemented our expanded SciDAC investments with activities at each facility dedicated to assisting “pioneer applications” that are important to the Department and poised to make good use of the leadership capability when available.  These applications are getting early access to the machines but also a lot of help from the machine vendor, facility staff, and SciDAC teams to improve performance and deliver more science.  The lessons learned in these activities will not only help those select applications but will be applied to the facilities’ other users and other SciDAC teams.

This is important because over the past five years we have opened our leadership computing facilities to all researchers, through a program called INCITE (for Innovative and Novel Computational Impact on Theory and Experiment).  This year we made a quarter of a billion hours of computing time available through INCITE, and the response has been wonderful.  Ever since we opened INCITE to all, in 2006, we have been able to offer an order of magnitude more hours each year, but the requests have always exceeded what we have available by about a factor of three, so the community really is moving just as rapidly as we are – including an increasing number of applications from industry.  Next year we will have a billion hours to award through INCITE.

Some of our INCITE projects will be presenting at the Masterworks session later this morning.  The industry projects tell a very interesting story.  Some turn to INCITE to inform their in-house purchase decisions, some have a very specific tests to run, while others have multi-year proposals for some very basic research that is focused on a real application and could deliver results for their products. 

The many facets of how high performance computing is contributing to U.S. competitiveness today is already obvious to us from our INCITE experience.  This validates the President’s vision, articulated in the American Competitiveness Initiative, that investments in basic research in key areas of the physical sciences and high performance computing are key to sustaining America’s competitive advantage.  This initiative will enable the Office of Science to continue to push the state of the art not only in facilities but in getting the most out of those facilities.

(click to enlarge slide)

Procter & Gamble

Bubbles and suds (aqueous foams) are ubiquitous in personal and home care products.  However, our only knowledge of surfactant assisted aqueous foam generation, growth and stability is empirical.  Understanding the molecular mechanisms of bubble formation, dynamics, and stability are important for transforming our knowledge (i.e., beyond incremental improvement) of sudsing detergents, but are also of interest for developing better fire control chemicals, chemicals for hazardous cleanup/remediation, as well as designing environmentally friendly consumer products.  The objective of this proposal is to gain insight into aqueous foam through large scale atomistic molecular dynamics simulations of cavitational and plateau regions of foams, and resultant coarse grained simulations of multiple dynamic, interacting bubbles.

Using INCITE, P&G has been able to run multi-million atom models of a pure, generic surfactant as a prelude to running a billion atom model of a real, commercial surfactant, or soap bubbles and foam.  This is the first time that P&G has been able to predict the fundamental nature of materials that it has not yet made.  Procter & Gamble is so confident that the billion atom model will be useful that it is making plans to incorporate the results into the development of real commercial products.  This would not have been possible without access to a system of the capability at Argonne.  And only DOE labs have this capability

Pratt & Whitney

Pratt & Whitney, a United Technologies Company is a world leader in the design, manufacture, and service of aircraft engines, industrial gas turbines, and space propulsion systems.  Every few seconds – 20,000 times each day – a Pratt & Whitney-powered airliner takes flight somewhere in the world.  Pratt &Whitney and its customers are constantly striving for green products with high fuel efficiency and low emissions.  The jet engine combustor is where fuel and air come together and it's critical to these goals, but is very difficult to test.  The flow inside a combustor is complex, faster than any hurricane, combusting, and at tens of atmospheres of pressure.  This means that computer simulation is essential to fielding competitive products, but simulations this complex have traditionally had cycle times that limited their potential in the design process.

Pratt & Whitney recognized that the DOE Leadership Class Computing resources offered a unique opportunity to explore the advanced combustor simulation techniques that next generation designs will require.  Through INCITE, Pratt & Whitney used a supercomputer at Argonne to push simulations far beyond the scale of Pratt & Whitney's previous practices to constructively "break" the existing simulation tools.  This enabled Pratt & Whitney to "remake" a process that can make use of 10X as many processors as the original and deliver results faster than ever before.

Pratt & Whitney has just announced the Geared Turbofan™ Engine, which will deliver game changing performance including double-digit improvements in fuel burn and emissions.  The capability improvements made through INCITE are being used right now on this revolutionary product.  Pratt & Whitney's INCITE team is continuing to push the envelope, focusing on larger and more accurate models.

Water

Understanding the structure of water in its many phases is fundamental to research in fields as diverse as biochemistry, cellular biology, atmospheric chemistry, and planetary physics.  While the properties of the bulk fluid are relatively well characterized, much less is known about water confined at the nanometer scale, where conventional experimental probes (neutron diffraction and X-ray scattering) are difficult to use.  This proposal will investigate water in confined states by (1) carrying out ab initio simulations for water confined between hydrophilic and hydrophobic surfaces and (2) studying the influence of dimensionality reduction and surface chemistry on the properties of the confined fluid.  The grand challenge is to define a computational paradigm to simulate water flow and transport at the nanoscale which can be applied to both materials science problems (e.g., water in zeolites) and problems of biological interest (e.g., water in contact with amino acids and proteins).

Using INCITE, the research team found “Sticky yet fast water at the interface”: perturbation induced by confinement is local; no ice-like layer at the interface –liquid density increases; rarefaction and decrease of density away from the interface; dipole moment of water molecule at the interface decreases* ->lateral diffusion is enhanced and re-orientational dynamics is faster – consistent with rapid flow in a nanotubes detected in recent experiments (*consistent with results obtainded for Benzene and HFB in water: M. Allesch, E. Schwegler and G. Galli. JPC-B 2007).

Parkinson’s Disease

Simulation and Modeling of Synuclein-Based 'Protofibril Structures' as a Means of Understanding the Molecular Basis of Parkinson's Disease (Igor Tsigelny, University of California -- San Diego/SDSC)

Recent studies suggest that the pathogenesis of Parkinson's Disease is related to the misfolding and abnormal accumulation of a-synuclein in neurons.  Although several NMR and biochemical studies have investigated the folding abnormalities of alpha-synuclein at fibrilizition state less is known about early alterations.  The molecular dynamics and docking studies conducted using an INCITE award and corresponding experiments of Dr. Tsigelny and Dr. Masliah have revealed the early folding anomalies of synuclein, suggesting that aggregates form porelike structures with channel activity.

Boeing

Through INCITE, Boeing is using the Oak Ridge Leadership Computing Facility to continue widening Computational Fluid Dynamics (CFD) use for the complete flight envelope.  Exploration of the design space will utilize more CFD to complement wind tunnel testing for upfront cycle time reduction during airplane design.   In addition to improving airplane performance and efficiency, it can also contribute to safety.

• Cut development and validation flow time to transition newer CFD tools for production
• Better and more accurate computational tools allow better, safer, and usable earlier in the design process
• Enable engineers to widen the application envelope which has not been much looked at before

(click to enlarge slide)

As we look further ahead, we see, and I am surprised to find that we really can say this with a straight face, that exaflop systems are on the horizon.  Just making them work will be a significant challenge, and these systems will be extremely complex, and many of the tools and techniques that we have worked so hard to develop may need to be re-invented or replaced.

In the Office of Science we take these challenges very seriously and are already investing in the long-term research that we hope will help meet those challenges, beecause with great risk comes great reward, and with these systems we once again will be on the verge of an important new era for science.

(click to enlarge slide)

We hosted three town hall meetings around the country to pulse the community about areas of science that need more computing capability to reach their goals.  The areas that were highlighted were climate modeling, all aspects of the sustainable energy production and delivery system, the life sciences and social sciences, which are really only just beginning to tap the power of computation, and at the opposite end of the spectrum, the very experienced community of astrophysics that includes high energy and nuclear physics as well.

Climate

How can we improve our understanding of complex biogeochemical cycles that underpin global ecosystems functions and control the sustainability of life on Earth?

• Improvements in representation of biogeochemical cycles in Earth System Models (ESMs) can be achieved by using a combination of data assimilation and development of mechanistic and process based models.
• Higher spatial resolution is needed to address the fine-scale heterogeneity inherent in biogeochemical process.
• New, innovative approaches are needed both in fundamental applied mathematics and in computational science to quantify the uncertainty inherent in a large systems-level model such as the ESM.

Energy

Providing new models and computational tools with the functionality needed to discover and develop complex processes inherent in a new energy economy. Three pathways to a low-carbon economy: computational nanoscience and materials science for renewable energy; simulation modeling for a fusion pathway: and simulation and modeling for advance nuclear energy systems.

• Computational Nanoscience and Material Science for Renewable Energy
  • Needs materials optimized for hydrogen storage
  • Needs reliable and efficient catalysis for water dissociation in hydrogen production
  • Needs cost-effective, environmentally benign, and stable material for efficient solar cells
• Advanced Nuclear Energy Systems
  • Spent fuel reprocessing is very complicated and requires a large number of different materials - multiple pathways must be considered
  • Waste streams must be treated
  • Improved coupling between computations and experiments must occur
• Fusion Energy - the promise of ITER
  • Designed to produce 500 million Watts of heat from fusion reactions for over 400 seconds with gain exceeding 10 – thereby demonstrating the scientific and technical feasibility of magnetic fusion energy
  • Fusion fuel will be sustained at high temperature by the fusion reactions themselves
  • Data from experiments worldwide, supported by advanced computation, indicate that ITER is likely to achieve its design performance

Life Sciences

Computational techniques are now enabling us to begin to understand this diversity through bioinformatics: reconstructing their genomes and identifying novel proteins; modeling their metabolisms; and modeling their complex multispecies communities.

• Model driven HT experimental data Generation
• Improving Model development
  • Genome scale metabolic networks, regulatory networks, signaling and developmental pathways
  • Microbial ecosystems and complex
  • biogeochemical interactions
• Bioinformatics techniques to address the integration of genomics, proteomics, metagenomics and structural data to screen for novel protein function discovery
• Molecular modeling techniques that can address the multiscale challenges

Astrophysics

More realistic (very high resolution) 3-D, multi-scale, multi-physics simulations will extend the frontier in many areas:

• Large Scale Structure Formation - Simulations of the large-scale distributions of galaxies and galaxy clusters over a large fraction of the observable universe with one percent precision, required of the observational program seeking to understand Dark Energy.
• Galaxy Formation – Simulations of galaxy formation with sufficient resolution to predict the observed properties of individual galaxies in a volume containing a sufficient fraction of the observable universe to compare with large-scale surveys.
• Stellar Evolution – Simulations of the entire stellar envelope in AGB stars, reasonable for the supply of half of the heavy elements in nature.
• Supernovae – Definitive 3-D multiphysics simulations of the core collapse supernovae, the dominant source of elements between oxygen and iron and the half of the heavy elements not produced in AGB stars.
• Compact Objects – definitive simulations of binaries involving two neutron stars and one black hole and one neutron star, which are among the leading candidates for the production of gravitational waves in our galaxy.

Social Science

Opportunities include the creation of coupled socio-economic-environmental models and greatly improved statistical analysis of data.

• Exascale computers have the potential to transform understanding of socio-economic environmental interactions through more detailed treatments of the various components and their interactions, and of issues of uncertainty and risk.
• Substantial effort must also be devoted to data quality and parameter estimation issues.
• Construction of a comprehensive suite of models of unprecedented geospatial and temporal detail with comprehensive error analysis on the representation.
• Leverage state-of-the-art climate modeling activities (e.g., SCIDAC) to include economic prediction models under alternative climate regimes.
• Basic research into spatial statistics, modeling of social processes, relevant microactivity and biosphere coupling issues, and relevant mathematical challenges, such as multiscale modeling.
• Assembly and quality control of extensive data collections.
• Comprehensive and detailed validation of both individual models and large model systems.
• Development of novel, robust numerical techniques and high-performance computing approaches to deal with the expected orders-of-magnitude increase in model complexity.
• A wide range of application studies aimed at both validation and application.

Education programs aimed at training the next generation of computational economists and other social scientists, including not only formal training programs but also web-based modeling and simulation tools that allow widespread access to the new models and their results.

(click to enlarge slide)

Then:  It has been the Office of Science position that 50 Teraflops sustained was the breakthrough speed, amounting to 125 Teraflops peak, for true scientific discovery using computational simulation.  We will break through that barrier literally within months.

Now:  We believe that in less than ten years, computers utilizing massive multi-core chips will achieve peak speeds 1,000 times a petaflop, or in these units, an exaflop speed.

Such speeds, while certainly unprecedented, also change the sociology of computation and indeed of science itself.  With such speeds, the scientist and the computer are no longer separable.

Effective use of exascale systems will require fundamental changes in how we develop and validate simulation codes for these systems and how we manage and extract knowledge from the massive amount of data produced.

• Exascale computer architectures necessitate radical changes to the software used to operate them and the science applications. The change is as disruptive as the shift from vector to distributed memory supercomputers 15 years ago.
• Message passing coupled with sequential programming languages will be inadequate for architectures based on many-core chips.
• Present code development, correctness, and performance analysis tools can’t scale up to millions of threads.
• Checkpointing (process for resuming code runs at an established “checkpoint” after a machine failure) will be inadequate for fault tolerance at the exascale.
• Fundamental changes are necessary to manage and extract knowledge from the tsunami of data created by exascale applications.

(click to enlarge slide)

The progress we have made has been important, but I believe that the real promise of simulation is yet to be tapped.  My vision for the future is to harness the capability that is swiftly becoming available and renew our commitment to make it available to all of science so that we can begin to realize the potential for simulation in policy and decision-making.

We have been using simulation through traditional approaches – using computers to perform calculations that we understand but could never complete in our lifetime.  However, the real promise of simulation lies in breaking free of the traditional scientific approach with its distinct disciplines, specializations, and reductionism.  The real promise of simulation is in integrating across the disciplines and exploring realms that are directly applicable to our lives – realms for which there are no equations, may not be much data, and where subtle connections result in hugely significant but currently unpredictable outcomes. This is what we are starting to call “systems of systems” research.

We are struggling to understand the impact of carbon emissions on the polar ice sheet via our traditional scientific approach.  With much more simulation capability and much broader, interdisciplinary thinking, might we learn how to connect the choices that each of us make to seemingly unconnected human issues such as famine, conflict and the spread of disease? 

This new approach may be taken in several key areas, which are actually interconnected.  

The first one is climate change – writ large to incorporate human factors of all kinds and incorporating the insights of our colleagues in the social sciences.

Second is natural resource management – energy, water, minerals, etc. with human factors and interconnections with climate models.

Third, the life sciences – from microbes and viruses to ecosystems including human factors both from the role we play in viral or microbial evolution and the impacts on human health.  To really be ready for the next flu pandemic, we need to understand epidemiology, viral evolution and vaccine development. We also need to understand the sociology, economics, and psychology of our health care system (including but not limited to vaccination), the economics of and politics of global cooperation, and possibly also volcanic activity.