Versioning Data Science Solutions

#1 in the Evolving Data Science Series

I come from an academic background, so when I talk about how versioning is done within data science, I speak from experience. However, over my career, I have honed my skills as a software engineer, architect, and tech innovation leader. This gives me a unique perspective on how things are and have been done as well as how they can be done better, much better.

Data science inherited its practice of versioning from academia. And this approach is what most any computationally scientific field has done for many years. Don’t get me wrong, computer science went this way in the early digital days (and in many cases, this is still prevalent in academia), but has moved onto more advanced practices.

What is this practice? It is the practice of creating ‘versioning’ schemes through ‘smart’ file naming. These schemes generally produce the following kind of files

This approach to versioning, which I refer to as faux versioning, is a disease and it is endemic. But breaking from these practices can be done, and should, as modern practices, such as model management, require it. This can be accomplished using a version control system (VCS)— with an industry preference for a git-based solution.

Using a git-based VCS with an appropriate branching strategy, you are able to immutably track the version of your ML models, and other artifacts in your data science development life cycle. But, the versioning of code, unlike with other software engineering disciplines, is necessary but not sufficient.

Additional issues must be considered:

While these additional concerns can be managed, a vanilla implementation will not suffice longer term and may cause more harm than help. Hashmap can help you down this path and put in place a solution that best fits your unique business needs.

This is part of the Evolving Data Science series.

Related posts:

One thing I love most in this world is its vastness, there are way more places than a person can visit in their lifetime, so the chase never ends, this is a journey without a destination, you can…

I recall a crazy summer night cruising around with my buddy Nate on the rural backroads that could have only been known as home. Nate and I had been drinking and my girlfriend (now wife) was the…

Sometimes being asked about your ethnicity can be perceived as microaggression especially in these times when race and anti-immigration issues are in the limelight. Yes, it may be out of genuine curiosity but may make the other person defensive if they think you are trying to judge them based on their geographical background.