Skip to content

AI-Translated Page

This page was translated using artificial intelligence (AI). The content may contain inaccuracies or misinterpretations. For critical or official use, please consult the site administrator to confirm the information.

August 30, 2021

MINUTES 003/2021 - Revision 002

Meeting of the Scientific Committee of the Unified Earth System Community Model. At 2:00 pm on August 30, 2021, representatives of INPE (National Institute for Space Research), INMET (National Institute of Meteorology), UFCG (Federal University of Campina Grande), CENSIPAM (Management and Operational Center for the Amazon Protection System), ITA (Technological Institute of Aeronautics), INPA (National Institute for Amazon Research), UFSM (Federal University of Santa Maria), USP (University of São Paulo), LNCC (National Laboratory for Scientific Computing), UFMS (Federal University of Mato Grosso do Sul), UFPA (Federal University of Pará), UFRJ (Federal University of Rio de Janeiro), MB (Brazilian Navy), FAB (Brazilian Air Force) and EB (Brazilian Army) met virtually with the aim of discussing and continuing the work of the Committee. Scientific Report of the Community Model of the Unified Earth System (MCSTU). These minutes record the memory of the meeting and gather the information inserted in the chat, such as links and information relevant to the discussions held. Following the opening of the meeting, carried out by Saulo Freitas, this document is oriented according to the guidelines established by Saulo Freitas during his presentation.

Opening

Saulo Freitas thanks all participants for attending the 4th Meeting of the Scientific Committee (CC) of the MCSTU, whose mission is to guide actions for the development of a community model of the Earth system. He comments that the MCSTU is a path to take the country to a new level of quality for weather, climate and environmental forecasts. The challenge is great and the expectation is that, with adequate resources, a functional model will be obtained that surpasses the forecasting capacity of the current models in use in the country within 5 to 10 years. Saulo Freitas then summarizes the agenda to be addressed at the meeting: a) introduction of new members and preliminary discussion of issues from the last meeting; b) process of choosing the definitive name of the community model of the unified Earth system; c) speech by the Coordinator of the CGCT (General Coordination of Earth Sciences) of INPE; d) review of the leading members of the subgroups and start of prospecting and reporting activities; e) report of meetings of the PAD (High Performance Processing) subgroup; f) brief tribute to Professor Saulo Rabello Maciel de Barros from IME (Institute of Mathematics and Statistics) of USP and collaborating researcher at CPTEC (Center for Weather Forecasting and Climate Studies) of INPE (deceased on 07/11/2021).

Saulo Freitas announces the new members of the CC of the MCSTU: Julia Clarinda Paiva Cohen/UFPA, Afonso de Moraes Paiva/UFRJ, Ricardo de Camargo/USP, Luciano Ponzi Pezzi/INPE, Celso Mendes/INPE, Hélio Abreu Nogueira/FAB and Luiz Claudio Oliveira de Andrade/EB. Then, he gives the floor to the new members to introduce themselves.

Julia Cohen thanks the invitation and reinforces the expectation of collaboration. Saulo Barros thanks and adds that local collaborations will be discussed and over time it will become clearer how the collaborations will be carried out.

Afonso Paiva greets all members and says that one of his motivations is to collaborate with everyone. He mentions that he works with ocean modeling and hopes to be able to collaborate with the development of MCSTU.

Ricardo Camargo thanks the invitation, comments on his experience with atmospheric and coupled modeling and on his expectations in collaborating with MCSTU.

Celso Mendes introduces himself, mentions that he is a retired civil servant, collaborator in a postgraduate course in modeling and thanks for the invitation.

Luciano Pezzi is grateful for the invitation, mentioning that he has experience with oceanic and atmospheric models, regional and coupled systems.

Agenda 1 - Review of Actions Defined in the Previous Meeting

Saulo Freitas reviews the actions of the last meeting, which are already underway. He mentions the request made for the preparation of an introductory letter to the MCSTU, which can be used by the leaders of the subgroups to present the model and form collaboration networks. He mentions the creation of a homepage to publicize the MCSTU and mentions that he is looking for resources to do this. He comments on the invitation to FUNCEME (Cearence Foundation for Meteorology and Water Resources) for collaboration in the development of the hydrology component of the MCSTU. Another subject mentioned is the management of software projects, given the complexity of the MCSTU.

Agenda 2 - Poll to Define the Name of the Community Model

Saulo Freitas introduces a poll to define the name of the MCSTU (whose current name and acronym are provisional). He comments that the idea is to conduct a poll in which all members can suggest names, which will later be put to a vote. All suggestions should be sent by email to Fabielle Mota (fabielle.mota at inpe.br), without copying to other people, with the following information: the name of the model in Portuguese and in English, the acronym, and the reasoning justifying why the chosen name is important to give a lasting brand to the community model. In a future meeting, Fabielle Mota will present the suggestions (without citing the authors) to the MCSTU CC, from which the choice will be made.

Agenda 3 - Speech by the CGCT/INPE Coordinator

Saulo Freitas invites Gilvan Sampaio and Viviane Campos to talk about the actions for financing the community model and its applications, as well as the formalization and assignment of the functions of the MCSTU CC.

Gilvan Sampaio thanks the invitation and summarizes the history of the latest actions related to the establishment of the MCSTU. He reports that he was in Brasilia on 3 occasions in July and August. There were meetings with several secretaries and 3 proposal letters were requested for the FNDCT (National Fund for Scientific and Technological Development): one for the BIG program (Georeferenced Information Base), another for the Biomas-BR program and another for the MCSTU (the letters were signed by Saulo Freitas, Pedro Dias and Gilvan Sampaio). Once the proposal for the MCSTU was authorized, Saulo Freitas, Pedro Dias and Gilvan Sampaio worked to draft and submit the proposal, which was processed. The group was asked to add the supercomputing infrastructure to the proposal. This was done and a revision of the proposal was sent with the title "Renovation of the INPE supercomputing infrastructure and its application to meet the growing demands of Brazilian society for better forecasts, monitoring of weather, climate and the environment". The proposal was very well received and Gilvan Sampaio believes that it is very likely that the funds will be obtained via FNDCT, via FINEP and the resources should be managed by the support foundation, FUNCATE . He mentions that approximately R$ 7 million per year were requested for a period of 3 years for the MCSTU. The amount consists of scholarships, payments to legal entities and consultancy. At the moment, the FNDCT's response is awaited, but he emphasizes optimism regarding the proposal based on all the negotiations. He mentions that the construction of the model justifies the updating of the infrastructure and vice versa. He announces that the quota of scholarships from the INPE PCI program (Institutional Training Program) was divided between the 3 programs, and with this, at this time, it was possible to launch 5 scholarships directly related to the MCSTU. He adds that there will also be a conversation with FAPESP (São Paulo State Research Support Foundation) about the MCSTU. In meetings with the MCTI (Ministry of Science, Technology and Innovation), with Secretary Marcelo Marcos Morales, it was requested that an ordinance be issued by the MCTI informing what the community model is, how it is being organized, among other information. In parallel, an ordinance by INPE will also be issued. These two ordinances will serve to formalize the functions of the CC of the MCSTU and, within the scope of INPE, the ordinance should be issued more quickly. Viviane Campos has already drafted this ordinance to officially invite the members of the CC of the MCSTU.

Saulo Freitas asked Gilvan Sampaio to clarify some more aspects of the leadership of the CC of MCSTU and the status of MCSTU as an INPE program. Gilvan Sampaio responded that MCSTU was conceived as a program associated with INPE, within which there may be several projects, but that despite this, it is also part of MCTI. Within the scope of MCTI, the CC of MCSTU should be led by Saulo Freitas and Pedro Dias. Within the scope of INPE, it should be an active employee, in this case Saulo Freitas, and Pedro Dias acts as a collaborator, which does not change his status in the coordination of the CC of MCSTU.

Saulo Freitas adds that, for legal reasons, the director of INPE cannot appoint or assign functions to Pedro Dias, and for this reason, Pedro Dias has the status of collaborator within INPE. Within MCTI, both researchers have the same status as coordinators.

Gilvan Sampaio adds that, regarding the financing actions, the scholarships can be distributed throughout Brazil, as decided by the MCSTU. He says that it is necessary to establish an official acronym for the program and that at the next meeting he would like to invite someone from the MCTI to participate in the meeting of the MCSTU CC.

Saulo Freitas thanks Gilvan Sampaio for the information and clarifications and gives the floor to the members of the MCSTU CC for comments and suggestions.

Enio Souza argues that, because MCSTU is an MCTI program, there is a transversality between several ministries ( Ministry of Agriculture, Livestock and Supply on behalf of INMET ; Ministry of Defense on behalf of the agencies involved; Ministry of Education on behalf of the universities, among others) and that this gives greater robustness to MCSTU.

Gilvan Sampaio adds that this transversality between the ministries was highlighted in the meetings with the MCTI. He mentions that in the construction of the SNM (National Meteorological System), there will be a decree, with the appointment of a management committee (formed by INPE , INMET and CENSIPAM ), but that the coordination will be in the SAE (Special Secretariat for Strategic Affairs of the Presidency of the Republic). In the construction of the SNM, the MCSTU will be very important. He adds that the idea of ​​transversality is being emphasized in all negotiations.

Luciano Pezzi questions whether the 3-year budget, with an amount of R$21 million, is only for the MCSTU, and how the update of the computing infrastructure will be addressed.

Gilvan Sampaio answers affirmatively and adds that the total budget requested is R$200 million, including the machine to make the model, the updating of the infrastructure (air conditioning, electrical energy, etc.). He informs that associated with this there is also the project to implement a first phase of a photovoltaic power generation plant in Cachoeira Paulista/SP, in addition to human resources.

Saulo Freitas adds that the focus of this R$7 million annual resource should be to finance the activities of the CC subgroups of the MCSTU for the development of the model throughout the country. External groups should be prioritized so that they have working conditions and contribute effectively to the development of the project.

Ronald Buss adds that this will be the first time that ocean and atmospheric numerical modeling will be worked on in a transversal manner within the MCTI. He comments on the lack of interaction between the MCTI sectors that work with ocean numerical modeling and the CGOA (General Coordination of Science for the Ocean, Antarctica and Geosciences) of the MCTI. An example of this is the REMO project , which is supported independently of the MCTI. He concludes that it is important to keep this type of situation in mind in order to avoid cooperation problems in the future.

Agenda 4 - Definition of Members of the Committee to Collect and Document the Requirements, Demands and Counterparts of the Unified Earth System Model

Saulo Freitas reinforces the idea of ​​the leaders of the subgroups of the CC of the MCSTU, reminding them who the members of the committees are, the counterparts and the demands. He adds that all the material generated by the subgroups will be organized by the CC of the MCSTU to generate white papers. Each appointed member will be the focal point of the respective theme and is expected to interact with the other partners of the national and international community. The members must document and bring for discussion at the CC of the MCSTU:

  • Requirements: in terms of physics, dynamics, computational grid, coupling technologies, PAD aspects, Data Assimilation methods, etc.;
  • Demands: applications, problems to be addressed, performance, accuracy, etc.;
  • Counterparts: in particular, what the Brazilian community can offer to international partners and institutions;
  • Infrastructure: resources for computing equipment and observation of the Earth system, data reception and processing, personnel, etc.

The material produced will be the basis of the model project to be forwarded to the competent sectors.

The members of the MCSTU CC appointed to the subgroups are:

  • Integrated Modeling System: Pedro Dias/USP

  • Atmosphere: Saulo Freitas/INPE

  • Continental Surface and Soils: Antonio Manzi/INPE

  • Ocean and Continental and Maritime Ice: Ronald Buss/INPE

  • Space Weather: Joaquim Costa/INPE

  • High Performance Processing and Code Quality: Luiz Flávio Rodrigues/INPE

  • Earth System Data Assimilation: João Gerd/INPE

  • Advanced Data Assimilation Methods and Artificial Intelligence Applications: Haroldo Campos Velho/INPE

  • Pre- and Post-processing Methods for Weather and Climate Forecasts: Caio Coelho/INPE and Carlos Bastarz/INPE

  • Surface and Subsurface Hydrology: FUNCEME/ IPH (Hydraulic Research Institute) to be defined

  • Organization of Demands and Perspectives of Universities and Research Institutes:

    • Northeast: Enio Souza/UFCG
    • North: Julia Cohen/UFPA
    • Southeast: Márcia Yamasoe/USP
    • South: Otavio Acevedo/UFSM
    • Midwest: Vinicius Capistrano/UFMS
    • CENSIPAM: Ivan Saraiva
    • INMET: Franciso Quixaba

  • Organization of the Demands and Perspectives of the Armed Forces:

    • Brazilian Navy: Flávia Rodrigues
    • Brazilian Air Force: Hélio Abreu Nogueira
    • Brazilian Army: Luiz Claudio Oliveira de Andrade

Saulo Freitas and Pedro Dias request that the subgroups begin to organize themselves and produce preliminary reports on how they are organizing and structuring the development of their themes. For the next meeting, Saulo Freitas and Pedro Dias will present what they expect for the atmospheric component, in terms of dynamic and physical cores. They also request that, for the next meeting, the initial organizational plans and perspectives with JEDI (Joint Effort for Data Assimilation Integration) for Earth System Data Assimilation (João Gerd/INPE) be presented.

Agenda 5 - Partnerships with PAS Companies - Report of Meetings with NEC and NVIDIA

Saulo Freitas comments on the meetings already held on the topic of High Performance Processing and Software Quality (Luiz Flávio Rodrigues/INPE). He requests that Luiz Flávio present his report on the meetings held.

Luiz Flávio presents a summary of the actions taken to create the code standardization document. He comments that, initially, this document is being written in Markdown language and that it is being versioned on the MCSTU Project Portal using the Subversion version control system (provisionally, since the goal is to use the Git version control system to be installed within the INPE computing environment). The document deals with software quality standards. He also mentions that forums were created on the MCSTU Project Portal to discuss the following topics: coding standards, testing methodology, development methods, and agile development management.

Luiz Flávio continues his presentation by reporting on meetings held with private sector companies NEC , IBM and NVIDIA . In addition to selling their products, these companies also provide PAD services. They are interested in forming partnerships that allow them to improve their products and services. In this sense, within the scope of the MCSTU, it is advantageous for the developments to be carried out due to the expertise developed by them. NVIDIA is a company that develops GPUs (Graphics Processor Units), a type of graphics co-processor, and it was responsible for porting the MPAS (Model for Prediction Across Scales) model to IBM (the IBM GRAF ). NVIDIA made two presentations about its developments and, in the first, Luiz Flávio highlights the pragmatism that NVIDIA had in not modifying the code too much, in order to reduce the impact of these changes on the researcher who developed it. In the second presentation by NVIDIA, examples were presented of the work done by the company on the dynamic core of the MPAS model and other models with which they worked. Regarding this presentation, Luiz Flávio highlights the differences between the examples of the original codes and the ported codes, with the use of OpenACC directives , so that the latter can be executed on GPU. In the examples presented (e.g., ICON , MPAS, FV3 , IFS ), the gains obtained are consistent (i.e., they can be observed in almost all the models worked on). Regarding the portability of the model codes to GPUs, Luiz Flávio concludes by saying that it is currently much simpler to make this transition. Regarding NEC's presentation, Luiz Flávio comments that the use of the new system developed by the company, TSUBASA (pluggable vector processing boards), which is a vector system, was presented, as well as the architectures of the NEC SX3 (1994-1998), SX4 (1998-2004) and SX6 (2004-2007), previously used by CPTEC before the transition to massively parallel machines, Sun cluster (2007-2010) and Cray XE6 (2010-2021) and XC50 (2018-) (1) , (2) , (3) , (4) , (5) , (6) . Luiz Flávio comments that NEC carried out tests with the BAM (Brazilian Atmospheric Model) models from CPTEC and WRF(Weather Research and Forecasting model), ported for use with TSUBASA boards, with which they obtained performance gains, which demonstrates the boards' ability to improve the performance of numerical models for weather and climate forecasting. Regarding the MPAS model, the company reported that it obtained gains of up to 2 times in the model's performance. Regarding the FV3 model, tests with TSUBASA boards are under development. Luiz Flávio emphasizes that the most important factor, in relation to NEC's solution, is the reduction in electricity consumption with the use of TSUBASA boards (significant reduction in the Watts/Flop ratio).

Luiz Flávio concludes the summary of the presentations by the PAD companies by stating that the dynamic cores and models discussed present good portability to different architectures (GPU), mainly due to the use of OpenACC. NEC's solution is interesting, as it opens the prospect of using a hybrid architecture machine, which should be considered in future acquisitions. The results presented by the companies show that there are significant improvements in the performance of the models using GPUs, in relation to the massively parallel versions of the models. He adds that it is important to have access to the architectures to validate the tests performed by the companies and to perform own tests with the dynamic cores to be selected for the MCSTU. He reports that Dell offered access to some models of machines that it manufactures and that it is healthy to keep the manufacturing companies close, in continuous contact with the MCSTU PAD subgroup, both for code development and for the exchange of information.

Saulo Freitas thanks Luiz Flavio for the presentation and asks about the average time that NVIDIA estimated to port the codes currently in use to GPUs, using the OpenACC directives.

Luiz Flávio, Pedro Dias and Saulo Freitas comment that, considering a cloud microphysics code with approximately 10,000 lines, the estimate is approximately 2 months. In this regard, the solution presented by NVIDIA still allows the code to be executed on massively parallel machines. Luiz Flávio adds that in the case of the BAM model, many parts of the model are written in pure Fortran language, which would facilitate portability, in addition to allowing these parts of the code to also be optimized. He also comments that some physics that NVIDIA ported for use with GPUs are also physics used by the model, which would already advance part of the work. Saulo Freitas adds that it is necessary to continue developing the models currently in use and acquire the necessary know-how on how to make this portability.

Saulo Freitas questions, within the scope of MCSTU, how a type of PAD support can be organized to support universities so that they can make use of these technologies.

Luiz Flávio argues that it is no longer an absolute truth to say that the academic environment is distant from PAD activities. He cites as an example the students of Professor Álvaro Luiz Fazenda /UNIFESP who have collaborated in some PAD activities at the center. Another aspect is the fact that the manufacturers of the machines are also interested in training PAD specialists. The MCSTU can be a channel between INPE and the universities, to train personnel in this area.

Pedro Dias comments on the familiarity with the use of GPUs. Nowadays, it is more common than before, but mainly in the area of ​​Computer Science. In other areas, such as Meteorology and Oceanography, it is more difficult to find many people familiar with the subject. At the IAG , an effort is being made to offer summer courses as complementary training for students. He encourages members of the MCSTU CC to expose their students to these new technologies and more modern processing architectures. He also comments on the need to have introductory material, seminars and classes on new technologies that can be distributed among the members of the MCSTU CC.

Luiz Flávio comments that, to a large extent, the current ease of porting codes is due to the evolution of compilers. He adds that it will be even more important for computer scientists to be closer to the scientists who develop the dynamics and physics of the models. This is necessary so that the initial developments are already prepared, for example, for the data structures appropriate to the computer architectures to be used.

Haroldo Campos agrees with Luiz Flávio and cites the example, within the scope of software engineering and project quality, of the telemetry software for satellite control developed by LAC . He also mentions the need for everyone to be able to participate in practical introductory mini-courses so that people can be informed about how to work in this type of programming. He recommends organizing a mini-course on heterogeneous computing.

Saulo Freitas agrees with Haroldo Campos and mentions the importance of organizing mobile workshops with PAD groups to provide on-site training and help solve problems. The premise is to allow the entire community to work on the MCSTU code. He adds that NVIDIA has committed to benchmarking the atmospheric BAM model (with prescribed ocean and ice), deterministic with 10 km horizontal resolution and with a uniform grid, for 10 days of forecasting, considering a 2-hour processing window. This benchmark, to be released in October 2021, will help understand what type of machine will be needed to carry out the MCSTU.

Pedro Dias comments that the interaction between computer scientists and model scientists must be two-way, ie, it is important that model scientists also communicate and explain how the parts of the model should communicate (in terms of the interactions between the different physical components of the Earth system).

Gilvan Sampaio comments that the MCSTU will be publicized and that there should be a seminar in the 2nd half of October 2021 at INPE.

Luciano Pezzi comments that it is essential for the PAD and applications groups to come together. It is essential to have the technical perception of the PAD team about what is feasible and what is possible to do with ocean modeling. Any decision to be made must consider what is already under development. Regarding couplers, there are also several and they are computational bottlenecks, he concludes that it is important to discuss them.

Pedro Dias argues that the topic of couplers is at the top of the list and is an important topic. It is necessary to identify who is interested in this subject and invite them to discuss it. Practical and theoretical issues are just as important, and time scales are an example of this. He mentions that to date, the ECMWF has not yet entered the non-hydrostatic modeling line, most likely due to coupling processes.

Haroldo Campos adds that the coupler is not only useful for the atmospheric and oceanic model, but also for the hydrological model.

Luiz Flavio informs that NVIDIA has ported MOM4 to GPU and therefore their experience also involves ocean models.

Enio Bueno comments that, regarding hybrid programming, it is necessary to think about the physics of the process. He does not believe that the model scientist should spend a lot of time learning to program on GPU, if the physics is not yet well defined and that it is necessary to spend more time to form the critical mass that will think about the physical problems. He concludes by saying that the technical problem of computing should be solved by computer scientists.

Luiz Flavio argues by citing the concept of pair programming (programming in pairs, between researcher and programmer). It is a paradigm shift in the numerical modeling community of the Earth system, which has a specific way of working. Agile methods can be adapted to this way of working.

Agenda 6 - Brief Tribute to Professor Saulo Rabello Maciel de Barros

Pedro Dias, Pedro Peixoto, Enio Bueno, Paulo Kubota, Luiz Flávio and Saulo Freitas share their life and professional experiences with Professor Saulo Barros and pay tribute to his memory and legacy.

Actions for the Next Meeting

  1. Poll to vote on the name of the unified earth system community model;
  2. Presentation of the perspectives for the atmospheric component of MCSTU (Saulo Freitas and Pedro Dias);
  3. Presentation of perspectives for Earth System Data Assimilation with JEDI (João Gerd);
  4. Brief communication from the other members of the MCSTU CC about the actions to organize the subgroups.
References
  1. INPE. News (SJC, 03/17/2006) - INPE supercomputer serves the country's scientific community http://www.inpe.br/noticias/noticia.php?Cod_Noticia=589
  2. INPE. News (SJC, 01/16/2007) - INPE purchases equipment with an eye on the next generation of CPTEC supercomputer. 2007. http://www.inpe.br/urc/noticias/noticia.php?Cod_Noticia=938
  3. CPTEC. History - Supercomputers - CPTEC/INPE. 2010. https://www.cptec.inpe.br/supercomputador/historico.php
  4. MENDES, C.; STEPHANY, S. Programming of Massively Parallel Systems. CAP Applied Computing - INPE. 2018. http://www.lac.inpe.br/~celso/cap396-2018/AULAS/Aula-2.pdf
  5. SOUZA, CR Evaluation of advanced MPI communication techniques in the execution of the BRAMS model. 2018. 108 p. IBI: <8JMKD3MGP3W34R/3R6QKR8>. (sid.inpe.br/mtc-m21c/2018/05.25.12.46-TDI). Dissertation (Master in Applied Computing) - National Institute for Space Research (INPE), São José dos Campos, 2018. Available at: http://urlib.net/rep/8JMKD3MGP3W34R/3R6QKR8
  6. STEPHANY, S. A brief history of supercomputing. CAP - Applied Computing - INPE. 2021. http://www.lac.inpe.br/~stephan/CAP-372/super2021k-linux.pdf
Attachments