How FogPharma leverages SciNote API and webhooks to automate data synchronization

Here’s how FogPharma, a Boston-based biopharmaceutical company, uses the API and webhooks of SciNote electronic lab notebook (ELN) to streamline data processing and enhance seamless integration and synchronization between various data systems.

“I like to think of systems as users, so when I’m evaluating a product like SciNote, it’s very important for there to be an API, because I want the system to be able to behave as if it were a user, and the more comprehensive that API is, the more opportunity there is to expand the scope of what that user can do…SciNote has a great comprehensive API”

Dave Veron, Principal Data Engineer, FogPharma

Summary:

• The challenge FogPharma faces was twofold: managing vast amounts of data efficiently and ensuring synchronized, up-to-date inventory without overloading the systems or causing disruptions.
• Compound inventory synchronization: The integration with SciNote ensured a parallelized and paginated data pull system. The efficient diffing system would detect and manage any changes or discrepancies between the two databases, ensuring synchronization.
• Data warehouse ingestion process: SciNote’s webhook functionality is expected to eliminate the need for constant polling by responding when an activity is logged, and eliminates the need for the user to notify the downstream systems that the event has occurred.
• Integrating SciNote into FogPharma’s operations has delivered critical enhancements in their procedural workflows, most prominently in managing data complexities and ensuring streamlined communication between different systems.

About FogPharma

FogPharma is a Boston-based biopharmaceutical company, with the mission to drastically reduce the burden of disease on patients and their families by inventing new types of drugs that block abnormal physiological processes in ways previously imagined but considered impossible.

Through the discovery and development of Helicons™, a novel therapeutic modality, FogPharma aims to address the limitations of existing drug classes and achieve universal druggability – a world where no targets are off-limits to new medicines.

FogPharma is a clinical-stage company led by an experienced team of scientists, R&D executives and company builders with a genuine passion to make a difference for patients through cutting-edge science.

The Challenge

Navigating the complexities of pharmaceutical research, FogPharma was presented with multiple challenges pertaining to data synchronization and management. The lab’s compound inventory and data warehouse systems required seamless integration.

For instance, the Compound Management Department would synthesize peptides, store and dispense them to researchers performing assays. However, there was a disconnect as there is a separate system registering the compounds at a virtual level and at a batch level.  The compound registry was frequently polled by the compound management system —every 15 minutes—to check for newly registered peptides among tens of thousands of records, making the process inefficient and potentially error-prone.

The need to continually pull data and the absence of a versatile public API at the time added layers of complexity. This is also the case when it comes to data warehouse ingestion.

The challenge was thus twofold: managing vast amounts of data efficiently and ensuring synchronized, up-to-date inventory without overloading the systems or causing disruptions.

The Solution

SciNote’s ELN emerged as an effective solution for FogPharma. With its comprehensive API, SciNote provided the ability for the system to act like a user, allowing for expanded functionality and improved workflow processes.

In terms of compound inventory synchronization:

• Jenkins, initially employed for job automation, was leveraged alongside AWS event bridge for cloud-based scheduling.
• The integration with SciNote ensured a parallelized and paginated data pull system. This method facilitated efficient data retrieval without overloading either system.
• The efficient diffing system would detect and manage any changes or discrepancies between the two databases, ensuring synchronization.

Regarding the data warehouse ingestion process:

• A planned integration using SciNote’s webhook functionality is expected to further streamline the system.
• This webhook will eliminate the need for constant polling by responding when an activity is logged, thereby making the process more dynamic. It also eliminates the need to modify the user interface to require an end user to press a button to inform the downstream systems that the event has occurred.
• The triggered system will ensure that new experiments and associated data get ingested seamlessly into the data warehouse, promoting real-time data handling.

Key Takeaways

The world of drug discovery is complex and multifaceted, characterized by intricate scientific processes, a vast amount of data, and the constant challenge of translating intricate research into tangible solutions for patients.

API Integration is Pivotal

“If a product doesn’t have an API, it tends to amount to a glorified spreadsheet. It ends up creating more work for my end users.”

As Dave points out, in modern research, the integration capability of systems isn’t merely a ‘nice-to-have’—it’s a necessity. Some key SciNote API features include:

Versatile Data Interaction: The API doesn’t just limit its users to simple tasks but provides various interaction methods. Users can delve deep into the SciNote system, accessing various data entities.

Uni- or Bi-direction Update: The API allows users to push updates back to SciNote, ensuring the system remains updated with the latest data. This bi-directional communication is crucial for real-time data management and accuracy.

Learn more about SciNote’s API system.

Webhooks provides real-time notifications

While APIs often require a user-initiated request to retrieve or send data, webhooks offer a more proactive approach. Here’s how they enhance the SciNote experience:

Instant External Notifications: Upon being triggered, the webhook immediately communicates with designated external systems. This could be another software tool, a messaging platform, or any other connected service.

This ensures these systems are instantly made aware of changes within SciNote, enabling faster response times and more dynamic integrations.

Configurable Interface: The beauty of webhooks within SciNote is their configurability. Users can specify which events should trigger a webhook and determine what data should be transmitted upon activation.

This means that each organization can tailor its webhook setup to cater specifically to its operational needs and preferences.

Learn more about webhooks in SciNote

Emphasis on Testing and Scalability

“You can think of API calls as triggering chain reactions. So if you have a mistake, you can expect that mistake 10,000 times; you don’t want that to happen in the production system.”

Diving directly into a production system with the API can lead to unintended consequences. Therefore, testing in a controlled environment is paramount.

By starting small and understanding the nuances of the API and its interaction with various platforms, organizations can then confidently scale up their operations, ensuring stability and efficiency.

Results

“[SciNote’s API] makes it simple, and makes it repeatable which takes user error out… it enables us to do exception handling and logging, it’s secure. It’s a great solution.”

Integrating SciNote into FogPharma’s operations has delivered critical enhancements in their procedural workflows, most prominently in managing data complexities and ensuring streamlined communication between different systems. This leads to:

• Enhanced API Integration: With SciNote’s robust RESTful interface, FogPharma has experienced more seamless and efficient API interactions, which simplifies the complexities inherent in system-to-system communications.
• Real-time Data Interaction: SciNote’s capability to interact directly with data entities has provided FogPharma with real-time access to relevant data. This has optimized decision-making processes and ensured that the most current data is always at the forefront.
• Proactive System Notifications with Webhooks: The addition of webhooks has been a game-changer. These complementary mechanisms to the API track changes within SciNote, and instantly alert external systems. This ensures data remains current without excessive pulling of information.

By implementing SciNote ELN and leveraging its RESTful API and webhook functionalities, FogPharma was able to drastically reduce manual oversight and improve workflow efficiency. Automated systems, when set up correctly, not only save time but also reduce potential errors stemming from manual data handling.