Auger Access

The Auger Access project is part of the Auger Observatory which is currently deployed on the Pampa Amarilla in western Argentina. The Auger Observatory is studying the universe's highest energy particles, which shower down on Earth in the form of cosmic rays. While cosmic rays with low to moderate energies are well understood, those with extremely high energies remain mysterious. By detecting and studying these rare particles, the Auger Observatory is tackling the enigmas of their origin and existence. To retrieve more information of the Auger project please visit the Auger Homepage.

The Auger Observatory is located in a remote region far from academic or research institutions. This is the main reason for the Auger Access project. The presence of scientists and technicians from the collaborating institutions on site is currently necessary during the phases of installation and commissioning of the detectors. In the long term, however, it will be very difficult to maintain such level of involvement at the site when scientists will be very busy with data analysis at their home institutions all over the world. It would also be exceedingly expensive to travel to Argentina every time an error occurred in the existing systems or to fix bugs in running software components.

The main goals for Auger Access are the remote monitoring and remote control functionality to provide access to the running experiment for the community worldwide. The monitoring and control functionality is necessary for developers and scientists to get status information about running components of the experiment. It is also intended to give the best possible support for the operators at experiment site in case of severe problems with the detectors. Of course the access to the data acquisition (DAQ) system for the “outside world” has to be handled very carefully. Unauthorized access has to be strictly prohibited. Now the access to the central campus in Malargüe for the “outside world” is blocked by a firewall in the central campus. Nobody can connect to the campus, only outgoing connections from the campus are allowed. To allow access to the project for the community methods have to be provided, that only authorized users can access. It is planned to use Grid Security to meet the access and security requirements.

Grid Security

To provide security goals a Grid services container must be installed at the central campus in Malargüe and the firewall must be opened for usage of the service container. Every user of the system needs a valid X.509 certificate from a Certificate Authority (CA). Worldwide there exists official CAs in every country signing personal certificates only after checking the identity of the potential users. The Grid services container at the central campus allows the access to the authorized users by comparing the users X.509 certificate and the known CA keys. If a user or a CA is unknown the request will be rejected and the access will be denied. Local administration efforts, e.g. user and CA administration, are necessary to realize this approach.

For the different user groups different security levels for the remote control/monitoring systems are necessary. For example only a small group of skilled system administrators needs to use the full remote control functionality. Other users would only be allowed to operate with the standard functionality.

Grid Middleware

To use the full functionality of a Grid a suitable middleware has to be found, which provides the full functionality of the Grid and is easy to extend. The middleware is although called as the Grid services container. There exist several middlewares with different approaches which provides the functionality of a Grid. First all these middlewares have to be checked to a suitable one for the given problem.

One component of most middlewares consists in the Grid Security. Another important component for the middleware is an easy way to expand the functionality it, necessary to extend the different remote monitoring and remote control tasks successively.

The usage of the new systems for the Auger collaboration is quite easy. After development of the individual systems there will exist a graphical user interface, which must be used to communicate with the several systems at central campus in Malargüe over the installed Grid services container. The users should not notice that they are performing operations in Malargüe, for them it looks as if they work on their local computers.

In fact it is a bit more complex when the user activates a function from the graphical user interface. First the user interface has to communicate with the Grid services container where the authentication and authorization of the user is done. After passing this step the Grid services container performs the chosen actions, in one extreme as remote control task. After receiving the necessary data, the services container pushs the data back to the graphical user interface, where it will be provided to the user. Now the user can work with the data or perform new functions by using the prepared interface.

The evaluation of the different Grid middlewares is completed and the Globus Toolkit 4 (GT4) in the actual release 4.04 is the chosen service container. GT4 is a wide spread, easy to extend Grid middleware which is developed by the Globus Alliance. GT 4 integrates libraries for several programming languages to make it easy to expand the functionality of the standard services container. It also ships with several tools to manage the container and its integrated services. For further information of GT4 look at the GT4 homepage.

Remote Monitoring

The remote monitoring should provide information about the state and the history of the system. The huge number of different subsystems to be monitored makes it extremely complicated to gather the necessary monitoring items. The performance monitoring database that is currently developed will provide a well-defined interface to the monitoring data. In close connection with the FD performance monitoring task this concept will be used to provide monitoring worldwide for the auger collaboration. The databases at the 4 Eye PCs in the 4 telescope buildings will be merged in the central campus. In every database log messages and data from slow control, data acquisition and calibration are stored. These messages are collected during run of the individual systems.

To minimize the needed space on local hard disks the database of every telescope building is replicated to a central database on the central campus holding the complete data. After replication the datasets on each Eye PC can be deleted. The integration of the databases in the existing system is shown in the first picture below.

The database approach provides access with a huge variety of applications and protocols, e. g. the operators at central campus in Malargüe can watch status messages and shut down systems on demand. After integrating the Grid Security, the access for the community to the database is possible, so that every registered user can watch the status messages available to the operators. Furthermore the operators can contact developers of the several systems to call in special expertise if needed.

Remote Control

The remote control is based on a sufficient remote monitoring. In case of severe problems, it is planned to have an option to give skilled system administrators the full remote control functionality. The full remote control of the system includes the starting and stopping of measurements, reboot of computers in error cases, opening and shutting the telescopes for data acquisition. The integration of the remote control in the existing system is shown in the first picture below.

To communicate with the remote control system the user firstly has to authenticate him and his privileges through the Grid services container. If this step is passed correctly the user can start using the system with the privileges he belongs to. When this are administrator privileges he could start or stop the system, reboot computers in error case and everything which is possible with the system. If the requested user is not known or has no privileges his request will be rejected and has no chance to interact with the system.

Architecture of the system

For the integration of new features, e. g. the remote monitoring or remote control it is necessary to integrate new components in the Auger experiment. One important component is the Grid services container, handling the authorization and authentication of users who want to interact with the project as mentioned above. Only with this service container and after authorization of every user it is possible to interact with the systems at central campus in Malargüe.

Another new component are the databases in the Eye PCs and the replicated database at operator side in Malargüe. The remote monitoring is only possible with these databases, because they hold all messages from data acquisition, slow control and calibration.

The other two new systems are the remote data analysis, which is very useful for a fast access to new measured dates from the project and the remote software maintenance, which is necessary for quick integration of new or corrected software versions in the existing systems, without traveling to Argentina.

The new systems are integrated in the existing structure and the existing software must be adapted, to enable the usage, e. g. for the data acquisition and slow control it is necessary to write the log messages to the databases, otherwise it is not possible to use the remote monitoring.

The picture below shows a systematic view of the integration of the new components in the existing system.

Systematic view of the structure of the new components in the existing system.

Testbed

For developing of the remote control system we need a testbed to test our new developed software components. For this testbed there exists two possibilities. On the one hand we could buy all the necessary hardware components and deploy them the same way as they are deployed in the running system in Argentina. Because of the very complex existing system, this possibility would be too expensive, although for the necessary administration efforts of every computer and the network hardware. So we decided to use the second possibility and deployed our testbed with virtual machines on only 1 dedicated server.

Therefore we acquired a IBM Blade Server with 2 Dual Core Xeons and 16 GB memory. On this server we installed VMware ESX Server as hyper visor for abstracting the existing hardware and deployed the computers for the DAQ system as virtual machines (VM). For every VM the necessary Operating System is installed and configured as in the real DAQ system in Malargüe. To simulate the network deployed in Argentina all virtual machines are connected through virtual switches, so they can communicate with each other as the real system does. At the moment the computers in Central Data Acquisition System (CDAS) for the interaction with the DAQ, Gina and Helge and the firewall computer for the communication with the world, are deployed. Although 2 telescope buildings (Los Leones and Los Morados) with EyePC, Calibration PC, Slow Control PC and 6 Mirror PCs each are deployed. Due to available resources on the ESX Server the remaining telescope buildings Loma Amarilla and Coihueco are not deployed.

In the picture below the VMs and their connection through virtual switches on the ESX server are shown.

Structure of our testbed with the individual computers and connection of them with network hardwaree.

Add Ons

Remote Data Analysis

At the moment all measured data from detected showers are stored in the central campus and are replicated to a server in Lyon, where the Auger collaboration can access and perform analysis. The replication of the big datasets to Lyon takes a long time, because of the unreliable and slow Internet connection from central campus to the world.

It is planned to provide a remote data analysis system at the central campus to allow the scientists immediate access to the data before replication. All dates are stored in ROOT and a ROOT script is necessary to perform analysis on the data. Scientist will be allowed to send ROOT scripts to a service within the Grid services container at the central campus and to receive the results of the performed calculations online.

Remote software maintenance

In the current state of the system a software developer has to travel to Argentina to install new software. Thus it is very time consuming to test new algorithms or to fix bugs in the existing software. After installing the Grid Security a method will be implemented to change the running software from all over the world. Therefore a developer can connect to the system, update the existing software version and start the new one. Before installing a new version a backup of the existing version will be made to switch back, if the new one does not have the right behavior.