This is the first of a recurring tech series chronicling the tasks I took on over the previous week at work. It is an opportunity for reflection, to keep track of the skills I have been honing, and to admire my own accomplishments and not-so-great moments alike.
System Integration & Sync
As two systems grow independently, the more they are likely to continue to diverge. Something something entropy something something chaos, right? Fortunately or not, having a bonafide business need to wrangle very different systems is not unrealistic, it is a fact of life. While the specifics of any integration task vary greatly, the common theme of allowing the flow of information to continue through the boundaries of disparate systems prevails. I can list two overarching challenges that make handling this kind of data difficult.
Challenge 1: Data is Temporal
Simply transferring data from one system to another is not always sufficient. How the acceptable time-to-live (TTL) threshold for a particular system’s copy of data varies. Some data is added to historical entries, and grow together as a collection, while other systems have data that is overwritten. The latter can be more challenging.
Challenge 2: Data flows through Business Processes
Beyond just having different natures of data evolution and tolerance for old data, there are issues when a business problem requires that data flows bilaterally between multiple systems but perhaps they are only synchronized unilaterally. Some processes may require accessing complementary data or the same data from different perspectives via two systems at once or back and forth, perhaps asynchronously rather than in a predictable, serial fashion. Perhaps it is difficult to conduct the interacting processes or processing speed needs require independent access to data. In any of these cases, making systems work together becomes considerably more difficult if access is not either centralized, controlled, or synchronized in each direction it needs to flow.
Real World Applications
Leaving the abstract behind, the above becomes less clear cut in the real world. Needs may change over time, or become apparent after work has begun to integrate systems. So called “academic problems” may have more stringent requirements whereas a business application may work around restrictions for the sake of practicality. Properly approaching the issue at hand with the system analysis and design approach can be very beneficial for discovery and scoping.
- Planning
- Analysis
- Design
- Implementation
Conducting interviews, gathering requirements and usage information, and other tasks can help make needs more concrete and give light to leeway you may have when facing technical limitations.
CRM Intregration
This week I specifically worked on a project to synchronize data between a proprietary CRM system and Salesforce.com — a unilateral data flow.
The Problem
– Data needs to stay up to date between both systems, the proprietary one having some flexibility and being considered the master data source.
Analysis
– The business needs of the data in salesforce dictate that an acceptable TTL for data is within 24 hours. Activities that rely on the data in salesforce can be triggered to occur after the sync’s scheduled time. Synchronizing after hours when data is no longer modified in the internal CRM allows the data to be collected when volatility is at its lowest. Using the data shortly after the synchronization when possible ensures that its effective TTL is much lower.
Design
– Whether to push or to pull data in a unilateral synchronization use case is an initial decision you need to make. In most cases, you’ll want to consider the flow of data relative to the source you have the most control over. In this case, this is the proprietary CRM as it is developed in-house. Control over when and how the synchronization is triggered is straightforward when software is deployed on the local system.
Implementation
– In this case I wrote a C# console app that is executed on a local server as a scheduled task. It pulls relevant data from a customized DB view, transforms it, and sends it in batches over Salesforce’s bulk API, where the processing itself is offloaded to their servers. Your (and mine) may involve multiple points where triggers must fire. Indeed, in this case, some records are submitted to both systems. Such records are linked at the time of their creation so they may be updated at synch intervals using their primary keys or other UUIDs that are shared between them. Records that are missing in the remote system can be upserted.
This is a challenging but gratifying project to work on with a large business impact. When considering such synchronizations, you need to consider whether to work in batches and have a higher TTL or to work as data is entered and as it changes, working with single records (or perhaps queuing changes and collecting them at short intervals). The former can work with less triggers and more independently with the system, while the other technique requires deper coupling and control over one of the systems.
Site Overhaul
This week I collaborated with several coworkers to overhaul an existing company website. Doing so was chosen as an alternative to entirely scrapping the site and carrying the content over. Given that I was extending and redesigning portions of the site, I had the satisfaction of having a lot of the work already out of the way. It also came with the extra need to ensure that design standards do not clash between the old and new sections of the site unless aspects across the whole site were changed. Inconsistency can wreak havoc on a project to (incrementally) update a site, especially if a lot of time elapses between design passes.
Throughout the time I worked on the overhaul last week (still in progress), I found some bugs with edge cases of certain code. While that did not derail the project, reusing old code allowed me to improve other places where the code was used. This site’s redesign was wireframed out ahead of time, giving a good starting point and set of expectations. The actual development, however, involved iterative changes with frequent check-ins from other people involved. Without an established wireframe, having other people involved too frequently or too early can be disruptive, but in this case it seemed to be helpful as it allowed preferences to be expressed early and often, before any work was done too far in a wrong direction.
Overall the past week was very busy, caffeine filled, and enjoyable.