As part of the openIMIS-coreMIS integration project, we migrated the reporting system from Elasticsearch and Kibana to OpenSearch. This transition ensures open-source compatibility and long-term sustainability while maintaining powerful search and analytics capabilities. In this case study, we share the key steps, challenges, and best practices from our integration process.

What is OpenSearch?

OpenSearch is an open-source search and analytics suite designed for handling large-scale data queries, real-time log analysis, and observability. Businesses use OpenSearch to power applications like log monitoring, full-text search, security analytics, and business intelligence dashboards. It provides a scalable way to store, search, and visualize massive datasets efficiently.

A key component of OpenSearch is OpenSearch Dashboards, a visualization and analytics tool similar to Kibana. It allows users to create real-time dashboards, monitor system health, and analyze trends in data – essential for companies relying on data-driven insights.

Why OpenSearch exists: The Elasticsearch Fork

Originally, many organizations used Elasticsearch, built on Apache License. In 2021, Elastic changed its license from open-source (Apache 2.0) to a more restrictive one, limiting free use. In response, AWS forked Elasticsearch 7.10 – the last open version – and created OpenSearch, along with OpenSearch Dashboards (a fork of Kibana), ensuring an open search platform remained available.

Key differences: OpenSearch vs. Elasticsearch

The business impact

For companies relying on search and analytics, OpenSearch offers a cost-effective, open, and scalable alternative to Elasticsearch. Organizations seeking freedom from vendor lock-in, community-driven development, and enterprise-grade features without additional licensing costs have increasingly adopted OpenSearch.

In April 2023, we began working on the integration of a social protection layer into openIMIS as part of the openIMIS-coreMIS merger project. The legacy coreMIS system, originally built on Java, utilized Elasticsearch and Kibana for reporting and analytics.

We’ve decided to transition to OpenSearch as the reporting tool for the migrated version. This decision was driven by the need for open-source compatibility, as OpenSearch’s Apache 2.0 license ensures greater flexibility and long-term sustainability without the restrictions imposed by Elasticsearch’s new licensing model.

Integrating OpenSearch into the openIMIS ecosystem using Docker compose

To set up OpenSearch within the openIMIS ecosystem, Docker Compose is used to manage and run OpenSearch services. The configuration is available in the openIMIS Docker distribution repository, where all necessary files for running the whole openIMIS package/distribution are included.

Environment configuration

A separate environment file, .env.openSearch.example, has been added to define key settings for OpenSearch. This makes it easier to configure and modify the setup when needed. The file includes:

These variables define the OpenSearch cluster name, ports, security settings, and authentication details. They are used within the Docker Compose setup to ensure a consistent deployment.

Docker compose configuration

The compose.openSearch.yml file includes the necessary services to run OpenSearch and OpenSearch Dashboards.

• OpenSearch service:
  • Runs the latest OpenSearch image.
  • Loads environment variables from .env.openSearch.
  • Disables security settings for easier local deployment. (DO NOT USE in production, only in DEV mode)
  • Includes a health check to verify that the OpenSearch cluster is running.

• OpenSearch dashboards service:
  • Runs OpenSearch Dashboards for visualization.
  • Connects to the OpenSearch service using the OPENSEARCH_HOSTS variable.
  • Disables the security plugin for easier setup. (DO NOT USE in production, only in DEV mode)

Configuration files

A separate folder, opensearch-configuration, contains important configuration files needed for OpenSearch. These include settings for indices, authentication, and dashboards to ensure OpenSearch works correctly within the openIMIS system. This setup allows openIMIS to run OpenSearch smoothly, making it possible to manage and visualize data in an open-source, scalable, and flexible way.

Embedding OpenSearch dashboards into the openIMIS frontend

After configuring OpenSearch with Docker, the next step was to integrate OpenSearch Dashboards into the openIMIS frontend. This setup allows users to access OpenSearch visualizations directly from the openIMIS web application.

Reverse proxy configuration with Nginx

To make OpenSearch Dashboards accessible within openIMIS, we added a reverse proxy configuration using Nginx. The frontend repository (openimis-fe_js) contains various Nginx configuration files, which define how different services, including OpenSearch, are exposed.

1. Main Nginx configuration (nginx.conf)

• This file includes other configuration files, such as openIMIS.conf, through the directive:

• It defines the server block that listens on ports 80 (HTTP) and 443 (HTTPS) for handling requests:

2. Container-based service discovery (var.conf)

• In Docker, services communicate using container names as hostnames. The variables file ensures that OpenSearch Dashboards can be reached under the correct container name:

• Here, opensearch-dashboards:5601 points to the OpenSearch Dashboards container, allowing seamless communication between services.

3. OpenSearch dashboards location configuration (opensearch.loc)

• To expose OpenSearch Dashboards under <OPENIMIS-HOST>/opensearch, the following location block was added:

This setup:

• Ensures authentication with auth_request /check_user/.
• Redirects traffic to OpenSearch Dashboards (proxy_pass http://${opensearch};).
• Forwards required headers to maintain secure communication.
• Uses Basic Authentication to secure access to OpenSearch Dashboards.

Embedding OpenSearch Dashboards in openIMIS using an IFrame

To display OpenSearch Dashboards within the openIMIS frontend, an iframe-based embedding approach was implemented. The module handling this functionality is openimis-fe-opensearch_reports_js. This approach allows OpenSearch visualizations and reports to be integrated directly into openIMIS UI, enabling users to access analytics without leaving the system.

Real-time synchronization between OpenSearch and openIMIS

The integration of OpenSearch with openIMIS enables real-time synchronization of data between different database systems. openIMIS primarily operates on PostgreSQL, while OpenSearch follows a NoSQL approach, similar to MongoDB. One of the challenges faced during this integration was adapting OpenSearch’s schema-less index management to work seamlessly with the structured ORM models in Django.

To facilitate this integration, openIMIS uses the django-opensearch-dsl library, which allows defining OpenSearch document structures based on Django ORM models. This approach ensures that whenever a database operation occurs (such as creating, updating, or deleting records in PostgreSQL), the corresponding data is automatically indexed in OpenSearch. 

The configuration for this synchronization is managed in the openIMIS backend assembly module, specifically in the opensearch.py settings file. Here, environment variables define OpenSearch host connections and authentication settings:

This setup ensures that OpenSearch remains updated with the latest changes from openIMIS. However, since some operations in openIMIS rely on raw SQL queries rather than Django ORM (for performance reasons, especially for bulk imports of individuals, groups, and beneficiaries), additional logic was required to manually trigger data synchronization.

Additionally, openIMIS now provides an option for users to enable or disable dashboard synchronization from the UI, as seen in the attached screenshots. This feature was introduced based on client requirements, allowing more flexibility in managing data updates across different OpenSearch dashboards. The ability to toggle synchronization helps optimize system performance by controlling which dashboards receive real-time updates.

The OpenSearch Dashboard UI itself is embedded within the openIMIS frontend via NGINX reverse proxy. Access to the dashboard is managed through the /opensearch path, with authentication handled by openIMIS. If a user is not authorized, Nginx returns the appropriate HTTP error. Once authenticated, the OpenSearch dashboard seamlessly integrates with openIMIS, enabling users to visualize and analyze data in real time.

Configuring OpenSearch documents in openIMIS Business Module

To integrate OpenSearch indexing with Django ORM models in openIMIS, follow these structured steps using the django-opensearch-dsl library. This allows seamless synchronization between structured PostgreSQL models and OpenSearch’s NoSQL index management.

Step 1: Create a documents.py file

Each business module in openIMIS that requires OpenSearch indexing should contain a documents.py file. This file will define how Django ORM models are represented as OpenSearch documents.

Step 2: Define an OpenSearch document for a Django model

Inside documents.py, create a Document class that represents a specific Django ORM model. For instance, to index Beneficiary data in OpenSearch:

Step 3: Define indexed fields and relationships

• Fields from Django model: The fields list specifies the columns from the Beneficiary model that should be indexed in OpenSearch.
• Foreign key handling: The related_models attribute ensures that associated tables (BenefitPlan and Individual) are included.
• Customizing field types: Instead of regular text fields, KeywordField is used for aggregable fields, and DateField ensures proper date storage.

Step 4: Register document for synchronization

The @registry.register_document decorator ensures that any create, update, or delete operation on the Django model triggers an automatic sync with OpenSearch.

Step 5: Optimize index management

• auto_refresh = True enables real-time synchronization. If performance issues arise, consider setting this to False and running bulk updates periodically.
• The queryset_pagination = 5000 setting improves efficiency when dealing with large datasets.

Best practices

1. Use KeywordField for aggregable fields instead of TextField, as OpenSearch does not allow aggregations on text fields by default.
2. Ensure auto_refresh is enabled unless performance constraints require manual indexing.
3. Maintain consistency with Django ORM by keeping field types in sync with the database schema.

How to export dashboards with related objects like visualizations in OpenSearch?

• Log in to your OpenSearch instance.
• Expand the sidebar located on the left side of the page.
• Navigate to Management and select Dashboards Management.
• On the left side of the page, click on Saved Objects.
• At the top-right corner of the table, click on Export <N> objects.
• Ensure that you have selected dashboards only. Additionally, choose the option to include related objects, and then click export all.
• You should have a file in .ndjson format.
• Save file in the business model for initialization after deployment in openimis-be_<module-name>/import_data.
• Rename filename into opensearch_<model-name>_dashboard.ndjson

How to import a dashboard

• Locate the dashboard definition file in .ndjson format within the openimis-be_<module-name>/import_data directory.
• Log in to your OpenSearch instance.
• Expand the sidebar located on the left side of the page.
• Navigate to Management and select Dashboards Management.
• On the left side of the page, click on Saved Objects.
• At the top-right corner of the table, click on Import.
• A new side-modal will appear on the right side of the page. Drag and drop the file from openimis-be_<module-name>/import_data into the import dropzone.
• This action will import the dashboards along with related charts that should be accessible on the visualization page.
• Verify if the dashboards have been imported properly.

Challenges and future enhancements in integrating OpenSearch with openIMIS

Challenges

1. Reverse proxy configuration with Nginx
   • One of the most significant challenges was integrating OpenSearch into the openIMIS ecosystem through Nginx reverse proxy.
   • This required extensive configuration to ensure proper request routing, security, and performance optimization.
   • A considerable amount of time was spent fine-tuning Nginx settings, especially handling authentication, CORS policies.
2. Limited documentation and community support
   • Unlike Kibana/Elasticsearch, OpenSearch has fewer case studies, tutorials, and shared experiences available online.
   • This made troubleshooting and best-practice implementation more challenging, as fewer references were available for similar integration scenarios.
3. Real-time OpenSearch synchronization with openIMIS
   • OpenIMIS and OpenSearch have differences in database structures and storage mechanisms, which pose difficulties in real-time data synchronization.

Planned improvements and future enhancements

1. Enhanced user access management
   • Improving authentication and authorization for the /opensearch path.
   • The openIMIS initiative is considering integrating Keycloak to manage user roles and permissions for OpenSearch access.
   • This would enhance security and allow role-based access control (RBAC) for dashboards and data exploration.
2. Advanced data pipelines and external system integration
   • Moving beyond real-time synchronization, openIMIS could introduce advanced data pipelines that allow for:
     • Bulk data processing and transformations.
     • Integration with external data sources and analytics platforms.
     • More efficient handling of large datasets.
3. Expanded dashboard management capabilities
   • Currently, users can edit existing dashboards, but adding new dashboards is limited.
   • A planned improvement is to enable dashboard creation directly from openIMIS, allowing users to define custom reports and analytics views without relying on manual imports.

OpenSearch as a BI solution: Conclusions and key takeaways

OpenSearch has proven to be a powerful business intelligence (BI) tool, offering a flexible and open-source solution for data exploration and visualization. It provides a user-friendly experience, enabling even non-technical users to analyze datasets once the system is properly configured. Additionally, its ability to investigate and optimize queries makes it a valuable tool for building insightful reports and dashboards.

However, despite these advantages, there are some challenges that organizations should be aware of:

1. Complex deployment and configuration
   • Integrating OpenSearch into openIMIS required extensive setup, particularly with Nginx reverse proxy and real-time data synchronization.
   • The process involved fine-tuning security settings, handling CORS issues, and ensuring smooth interaction between openIMIS and OpenSearch’s NoSQL-based indexing.
2. Community support and issue resolution
   • OpenSearch, being a relatively newer alternative to Kibana/Elasticsearch, has less documentation and case studies available.

Final verdict: A viable choice with room for improvement

Despite these challenges, OpenSearch has demonstrated its value and efficiency in handling business intelligence tasks within openIMIS. The experience gained from this project will make future deployments smoother, reducing setup complexity. 

With ongoing improvements, better user access management (e.g., Keycloak integration), and expanded data pipeline capabilities, OpenSearch can become an even more robust reporting and visualization tool for openIMIS and similar projects.