I have been working on The DIRAC Project as a part of my Google Summer of Code 2019 project since the last 3 months and it has been an amazing journey throughout which I would like to explain through the work I completed during this period.
Now I would like to showcase the changes I made to the code base in accordance with the tasks:
1) Setup DIRAC locally:
This was a challenging task for me as I was not much familiar with the DIRAC codebase and setup and I was not knowing the system’s internal working, meaning how to run services, how to stop them, etc.
So while working on this I came up with a Docker-Compose setup for DIRAC which was liked by my mentors and it turned out to be very helpful for development purposes and till date throughout my project period I used this setup. This setup contains three different containers namely one for DIRAC code and other dependencies, one for ElasticSearch, and the other for MySQL database and all these containers can interact with each other and we can also use volume mounts for changing the code.
This was my first contribution to DIRAC and the Pull Request on this can be found here which was merged.
2) Wrote Technical documentation on the gMonitor object:
In this task, I had to write technical documentation on DIRAC’s gMonitor object which uses Round Robin Database tool to monitor activities of various components within DIRAC.
The work on this can be found in these commits inside this Pull Request which has been merged.
3) Added ElasticSearch support to DIRAC’s Service component:
Starting with this task, I had to extend an existing ElasticSearch type named ComponentMonitoring which had few fields already being monitored. So based on studying DIRAC’s Service component I extended this type and added new key fields and monitoring fields.
Then next in this task, I had to migrate entire Service component such that we can configure it using a flag and users must be allowed to use ElasticSearch or gMonior object based monitoring for which I used DIRAC’s Configuration System and made everything flag enabled.
After this I started working on extending the existing plotter named ComponentMonitoringPlotter and added the new fields so that we can create various plots on these fields using the WebAppDIRAC which is DIRAC’s WebApp module.
Then I created a regression test for this plotter which was used to enure the proper working of the plotter.
This is a sample plot on usage of File Descriptors by various DIRAC Services:
Below is the list of commits made which are based on the above described work and inside this Pull Request:
1) Commits on extending the ES type and migrating the Service component can be found here in commit1 and commit2.
2) Work on the plotter is inside this commit.
3) The regression test was added in this commit.
4) Performance test:
In this task, I ran the existing performance test which is based on multi-mechanize and ran it once with monitoring based on gMonitor object and once based on ElasticSearch.
The results and notes on this can be found at this Google Docs link.
5) Worked on a use case and did migration of DIRAC’s Agent component to ElasticSearch:
First, I worked on solving a use case about which can be found here. In this use case, basically I had to work on sending the response time of a service to the ElasticSearch backend.
Work on this use case can be found inside this commit and I also added a plot for this new field which can be found in this commit.
Then I worked on migrating the AgentModule to ElasticSearch and the work on this can be found in this commit.
Apart from this I worked on this small issue where I had to make the MonitoringReporter thread safe which is used to send data to the ElasticSearch backend, the work on this can be found here in this Pull Request.
6) Added ElasticSearch monitoring support to RequestManagementSystem and DataManagementSystem/Agent/RequestOperations/:
In this task, first I had to study the use cases of gMonitor object inside RequestManagementSystem and DataManagementSystem/Agent/RequestOperations/ and then I had to work on designing an ElasticSearch type such that all the use cases of gMonitor object are encompassed which was a challenging task for me and I took some time in this but was later able to design it with inputs from my mentors.
After creating this ElasticSearch type I started migrating the two systems to support the new system and again made things flag configurable and also created a new plotter for this new type.
A sample monitoring plot after migration to ElasticSearch:
Some work on this is in progress and all the work till now can be found in this Pull Request.
Summarising things, all my work at DIRAC can be found in these PRs:
1) Adds Docker-Compose setup for development purpose #4085.
2) Adds ES monitoring support to services and agents #4120.
3) Makes the MonitoringReporter thread safe #4194.
4) Adds ES monitoring support to RequestManagementSystem and DataManagementSystem/Agent/RequestOperations #4209. [Yet to be merged]
Lastly, I would like to thank my mentors Christophe Haen, Zoltan Mathe, and Federico Stagni for their constant support, inputs, and help throughout the period. I got to learn a lot of things from them and they also helped me highlight my weaknesses and mistakes I made during the period. Overall, it was a very good experience working on The DIRAC project and I even look forward to contribute to DIRAC after the GSoC period ends.